You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Harsha <st...@harsha.io> on 2017/08/01 17:27:32 UTC
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes
Trying to bring attention this again.
We currently have few big feature PRs going on and there is considerable
discussion about the design and Implementation etcc. My intention of
starting SIP is to add these details before someone goes and writes up a
PR and everyone has to go through reading of design and sometimes those
docs are not clear and we end up having long discussion on the PRs which
should mainly about the code review itself.
We should at least start making this process mandatory for any new big
feature especially to the storm-core. I am less concerned about
connectors and other parts which should have least resistive path and
they are usually easy to review.
If the devs put their thoughts and design and goes through discussion
and get everyone on the same line when the PR shows up it will be less
surprising and everyone involved know how the PR/Code supposed to work.
-Harsha
On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
> Arun,
> For big features we did follow design doc/review. Making it
> formal makes everyone to follow a process.
> Again this process is not for bug fixes as we stated its about New
> Features/Config Changes/Public interface changes. I don't think it puts
> any extra effort for anyone who is writing detailed JIRA but by making
> it formal makes everyone to add these details in a centra process. Not
> everyone will look at mailing list but its easier to follow a wiki page.
> We should atleast give it a try before we vote it out.
>
> Roshan,
> Adding connector should require a SIP as well and changing any
> public interfaces should be a KIP. Intention here is we've
> central place where everyone can follow in detail whats the
> public interface/new feature changes went in. We've changed
> KafkaSpout quite a bit and there is current discussion thats
> going to change it , having this documented in a central place
> will make it easy to follow and recording them in release notes
> as well.
>
> Taylor,
> We can't call it a too tedious process without even giving it a
> try. This has been followed to a greater success at kafka and
> also Flink started the process as well
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> .
> If it actually proved to more of hindrance than helping the community we
> can move away from it.
>
> " Kafka has somewhat of a reputation for setting potentially too high a
> bar. I'd rather not see that happen with this community."
> Sure. But it also depends on the community. Just because some community
> enforcing too high bar that doesn't mean we are trying to do it via this
> process. Again we always have option if we ever veer too far in the
> wrong direction to bring up and improve or remove this process.
>
> We should also as a community strive to have better quality and I am
> hoping this will give us a chance to not only let users know what are
> changes coming in but also keep the dev list to have a chance and join
> the discussion.
>
> -Harsha
>
> On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
> I am for documenting and upfront design reviews, but maybe we should
> keep it less formal and make it part of the JIRA to start with.
>
> Do we have any upcoming features for which we would like to see a
> proposal? May be start with a couple of proposals
> and see it works out before making it formal.
>
>
> Thanks,
> Arun
>
>
>
> 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>
> -0
>
> The KIP process feels kind of heavy. I'd rather start with a lighter
> effort like improving JIRA submissions and pull requests (some pull
> requests/JIRAs, even from committers and PMC members, are woefully
> inadequate in terms of detail), and see how that works out.
>
> I share Bobby's concern that doing so might raise the bar for
> contributions and potentially have a chilling effect. We don't want to
> scare away contributors. Kafka has somewhat of a reputation for setting
> potentially too high a bar. I'd rather not see that happen with this
> community.
>
> I will say that I like the idea of proposals for big features, ideally
> before any coding even begins -- so that others have a chance to
> collaborate. But I'm hesitant to impose too much process, voting, etc.
> That could scare people off.
>
> I think we should think carefully before going down this trail.
>
> -Taylor
>
> On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
>
> +1 for SIPs including a new connector. The person writing SIP can
> provide details about the external system for which connector is being
> written to let others know why a certain design decision was made. This
> will make it easy for reviewers.
>
> On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
>
> +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
> would be very useful for new contributors and others who are looking out
> for a feature design and decisions taken etc.
>
> Whenever a minor feature is added to a connector it may not need a
> separate
> SIP but the existing README can be updated with details for users. It
> can
> be discussed and decided apropos whether a SIP is really required for
> any
> enhancement which is not really big.
>
>
> On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
> If I am looking at the Kafka site correctly, I see that Kafka has a
> total
> of 167 KIPs so far.
> So I assume that minor new features would not be parrt of the SIP ?
>
> Unlike Kafka, since Storm has a number of connectors (that keep
> growing),
> I am speculating the SIP process might get somewhat unwieldy if it were
> to
> track little changes in each of the connectors.
>
> Also thinking that a SIP may not be needed to justify a new connector,
> but
> useful if we are replacing an old connector with a new one.
>
> -roshan
>
>
>
> On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
>
> Hi Bobby,
> In general, a KIP is required for adding New features,
> config
> changes or backward-incompatible changes. Don't require
> adding a KIP for bug-fixes. Devs who wants to add any
> features will write up a wiki which has JIRA link, mailing
> list discussion link and outline the Motivation, Public
> interface changes and protocol changes etc ..a good example
> here is
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 48+Delegation+token+support+for+Kafka.
> They can start the discussion thread once its ready and once everyone
> agrees its in a good shape, a Vote thread starts . Once there are
> required votes are in one can start the PR process and get it merged
> in.
> Each release we can collect what features/fixes especially
> to
> public interfaces that went in and roll it out in release
> notes. This will give a better idea for the users on what
> changed and added from previous version.
> We can only enforce this to new feature/config/backward
> incompatible change. Having this go through the discussion
> phase will give us the early feedback and potentially caught
> any issues before the implementation.
> Thanks,
> Harsha
>
> On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
>
> wrote:
>
> Can you please explain how KIP currently works and how you would
> like to see something similar in storm?
> If we make the process more formal we will probably have less
> people
> contributing, but we will probably have better overall patches. It
> is a balancing act and having never used KIP I would like to
> understand it better before going all in on it.
> - Bobby
>
>
> On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
> <ge...@gmail.com> wrote:
>
> This sounds like a good idea. KIPs seem to work well for Kafka.
> It's
> easy
> for discussions to get lost or just not seen on the mailing list.
>
> 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
>
> Hi All,
> We’ve seen good adoption of KIP approach in Kafka
> community
> and would like to see we adopt the similar approach for
> storm
> as well.
> Its hard to keep track of proposed changes and mailing list
> threads to
> know what all changes that are coming into and what
> design/backward
> incompatible changes being approved. It will be good to have
> this
> documented and go through discussion then Vote phase to get them
> approved before we merge the PRs. This will keep everyone
> informed of
> what changes happened even if they are not following the mailing
> list
> they can go to wiki to see the list of changes went into a
> release.
> Community overall will be well informed of the changes as well.
> Would
> like to hear your thoughts.
>
> Thanks,
> Harsha
>
>
>
>
>
>
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss
changes
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I would advise against codifying a SIP process in our bylaws, at least not without doing an informal experiment to see how it goes, work out the kinks, etc. Once something goes into the bylaws, it can be very difficult to change, remove, etc.
I prefer operating more on general consensus versus hard and fast rules. It makes it easier to test new processes and course correct if things go sideways.
-Taylor
> On Aug 2, 2017, at 10:46 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> My only concerns are
> 1. It is going to add more overhead and slow down getting a feature out. But that may not be the case if there is less overhead during the review process to offset writing up a large design, and it can be balanced with small features not needing it. 2. that I am going to miss a SIP review because I am busy and then when I complain about something wrong with a feature during a code review I will get a lot of push back because it passed SIP.
> but I can also see both sides of the coin on this. To me SIP needs to be a change to the bylaws, so if you want this change Harsha I would suggest that you write up your proposed changes to the bylaws, and we can discuss them in more detail, because it feels rather abstract to me right now. And deciding what is a large feature vs a small one also feels a bit abstract.
>
>
> - Bobby
>
>
> On Tuesday, August 1, 2017, 6:30:56 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Still +1 to introduce this only for non connectors. Maybe would want to
> skip this also for non connectors and non storm-core
> (storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
> still have small chance to need it.
>
> Despite that I voted to +1, I still worry about efforts on reviewing SIP:
> this will only work if we (in dev@ list) are open to participate and review
> SIP in desired duration. Two sides of the coin: it might incurs more active
> community, but it will just fail if community is not enough active. Each
> SIP discussion is easy to be staled if we don't care about much, and if we
> also want to introduce vote for SIP, easier to be staled.
>
> So we all should have willing to go with this. I'm OK to take additional
> load, but would like to hear others opinions as well.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
>
>> Trying to bring attention this again.
>> We currently have few big feature PRs going on and there is considerable
>> discussion about the design and Implementation etcc. My intention of
>> starting SIP is to add these details before someone goes and writes up a
>> PR and everyone has to go through reading of design and sometimes those
>> docs are not clear and we end up having long discussion on the PRs which
>> should mainly about the code review itself.
>> We should at least start making this process mandatory for any new big
>> feature especially to the storm-core. I am less concerned about
>> connectors and other parts which should have least resistive path and
>> they are usually easy to review.
>> If the devs put their thoughts and design and goes through discussion
>> and get everyone on the same line when the PR shows up it will be less
>> surprising and everyone involved know how the PR/Code supposed to work.
>>
>> -Harsha
>>
>> On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
>>> Arun,
>>> For big features we did follow design doc/review. Making it
>>> formal makes everyone to follow a process.
>>> Again this process is not for bug fixes as we stated its about New
>>> Features/Config Changes/Public interface changes. I don't think it puts
>>> any extra effort for anyone who is writing detailed JIRA but by making
>>> it formal makes everyone to add these details in a centra process. Not
>>> everyone will look at mailing list but its easier to follow a wiki page.
>>> We should atleast give it a try before we vote it out.
>>>
>>> Roshan,
>>> Adding connector should require a SIP as well and changing any
>>> public interfaces should be a KIP. Intention here is we've
>>> central place where everyone can follow in detail whats the
>>> public interface/new feature changes went in. We've changed
>>> KafkaSpout quite a bit and there is current discussion thats
>>> going to change it , having this documented in a central place
>>> will make it easy to follow and recording them in release notes
>>> as well.
>>>
>>> Taylor,
>>> We can't call it a too tedious process without even giving it a
>>> try. This has been followed to a greater success at kafka and
>>> also Flink started the process as well
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>> .
>>> If it actually proved to more of hindrance than helping the community we
>>> can move away from it.
>>>
>>> " Kafka has somewhat of a reputation for setting potentially too high a
>>> bar. I'd rather not see that happen with this community."
>>> Sure. But it also depends on the community. Just because some community
>>> enforcing too high bar that doesn't mean we are trying to do it via this
>>> process. Again we always have option if we ever veer too far in the
>>> wrong direction to bring up and improve or remove this process.
>>>
>>> We should also as a community strive to have better quality and I am
>>> hoping this will give us a chance to not only let users know what are
>>> changes coming in but also keep the dev list to have a chance and join
>>> the discussion.
>>>
>>> -Harsha
>>>
>>> On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
>>> I am for documenting and upfront design reviews, but maybe we should
>>> keep it less formal and make it part of the JIRA to start with.
>>>
>>> Do we have any upcoming features for which we would like to see a
>>> proposal? May be start with a couple of proposals
>>> and see it works out before making it formal.
>>>
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>> 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>>
>>> -0
>>>
>>> The KIP process feels kind of heavy. I'd rather start with a lighter
>>> effort like improving JIRA submissions and pull requests (some pull
>>> requests/JIRAs, even from committers and PMC members, are woefully
>>> inadequate in terms of detail), and see how that works out.
>>>
>>> I share Bobby's concern that doing so might raise the bar for
>>> contributions and potentially have a chilling effect. We don't want to
>>> scare away contributors. Kafka has somewhat of a reputation for setting
>>> potentially too high a bar. I'd rather not see that happen with this
>>> community.
>>>
>>> I will say that I like the idea of proposals for big features, ideally
>>> before any coding even begins -- so that others have a chance to
>>> collaborate. But I'm hesitant to impose too much process, voting, etc.
>>> That could scare people off.
>>>
>>> I think we should think carefully before going down this trail.
>>>
>>> -Taylor
>>>
>>> On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
>>>
>>> +1 for SIPs including a new connector. The person writing SIP can
>>> provide details about the external system for which connector is being
>>> written to let others know why a certain design decision was made. This
>>> will make it easy for reviewers.
>>>
>>> On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
>>>
>>> +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
>>> would be very useful for new contributors and others who are looking out
>>> for a feature design and decisions taken etc.
>>>
>>> Whenever a minor feature is added to a connector it may not need a
>>> separate
>>> SIP but the existing README can be updated with details for users. It
>>> can
>>> be discussed and decided apropos whether a SIP is really required for
>>> any
>>> enhancement which is not really big.
>>>
>>>
>>> On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
>>> wrote:
>>>
>>> If I am looking at the Kafka site correctly, I see that Kafka has a
>>> total
>>> of 167 KIPs so far.
>>> So I assume that minor new features would not be parrt of the SIP ?
>>>
>>> Unlike Kafka, since Storm has a number of connectors (that keep
>>> growing),
>>> I am speculating the SIP process might get somewhat unwieldy if it were
>>> to
>>> track little changes in each of the connectors.
>>>
>>> Also thinking that a SIP may not be needed to justify a new connector,
>>> but
>>> useful if we are replacing an old connector with a new one.
>>>
>>> -roshan
>>>
>>>
>>>
>>> On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
>>>
>>> Hi Bobby,
>>> In general, a KIP is required for adding New features,
>>> config
>>> changes or backward-incompatible changes. Don't require
>>> adding a KIP for bug-fixes. Devs who wants to add any
>>> features will write up a wiki which has JIRA link, mailing
>>> list discussion link and outline the Motivation, Public
>>> interface changes and protocol changes etc ..a good example
>>> here is
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 48+Delegation+token+support+for+Kafka.
>>> They can start the discussion thread once its ready and once everyone
>>> agrees its in a good shape, a Vote thread starts . Once there are
>>> required votes are in one can start the PR process and get it merged
>>> in.
>>> Each release we can collect what features/fixes especially
>>> to
>>> public interfaces that went in and roll it out in release
>>> notes. This will give a better idea for the users on what
>>> changed and added from previous version.
>>> We can only enforce this to new feature/config/backward
>>> incompatible change. Having this go through the discussion
>>> phase will give us the early feedback and potentially caught
>>> any issues before the implementation.
>>> Thanks,
>>> Harsha
>>>
>>> On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
>>>
>>> wrote:
>>>
>>> Can you please explain how KIP currently works and how you would
>>> like to see something similar in storm?
>>> If we make the process more formal we will probably have less
>>> people
>>> contributing, but we will probably have better overall patches. It
>>> is a balancing act and having never used KIP I would like to
>>> understand it better before going all in on it.
>>> - Bobby
>>>
>>>
>>> On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
>>> <ge...@gmail.com> wrote:
>>>
>>> This sounds like a good idea. KIPs seem to work well for Kafka.
>>> It's
>>> easy
>>> for discussions to get lost or just not seen on the mailing list.
>>>
>>> 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
>>>
>>> Hi All,
>>> We’ve seen good adoption of KIP approach in Kafka
>>> community
>>> and would like to see we adopt the similar approach for
>>> storm
>>> as well.
>>> Its hard to keep track of proposed changes and mailing list
>>> threads to
>>> know what all changes that are coming into and what
>>> design/backward
>>> incompatible changes being approved. It will be good to have
>>> this
>>> documented and go through discussion then Vote phase to get them
>>> approved before we merge the PRs. This will keep everyone
>>> informed of
>>> what changes happened even if they are not following the mailing
>>> list
>>> they can go to wiki to see the list of changes went into a
>>> release.
>>> Community overall will be well informed of the changes as well.
>>> Would
>>> like to hear your thoughts.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>>
>>>
>>>
>>>
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss
changes
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
> On Aug 2, 2017, at 4:31 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> You don't want it in the bylaws yet, Not a big deal, but I still think having the rules formally written up are a good start, even if they are not formally voted on and binding in the bylaws.
> We can try it out see what works and what doesn’t.
Exactly. Essentially “try before you buy.” That way if we decide we don’t like it, we don’t need a 2/3 PMC majority to roll it back.
Tangent: There is also the possibility that certain proposals be summarily dismissed at the *idea* phase. This sort of happened with flux [1]. It wasn’t until the feature was fully implemented — outside the storm community — that the idea finally gained acceptance.
Granted, that’s only one isolated example. Some things about the KIP process feel waterfall-ish to me, and I don’t like the possibility of ideas being dismissed before given a chance (e.g. not being articulated well enough, non-native language, etc.).
That being said, I support trying it out as an experiment. But I agree with Bobby that the language of the policy needs to be worked out.
-Taylor
[1] https://issues.apache.org/jira/browse/STORM-561?focusedCommentId=14243304&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14243304
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss
changes
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
You don't want it in the bylaws yet, Not a big deal, but I still think having the rules formally written up are a good start, even if they are not formally voted on and binding in the bylaws.
We can try it out see what works and what doesn't.
- Bobby
On Wednesday, August 2, 2017, 12:29:46 PM CDT, Hugo Da Cruz Louro <hl...@hortonworks.com> wrote:
I am +1 with having a bit more structure in the process (SIP), but being cautious about limiting the overhead. I like the SIP idea, but I agree with Bobby’s point of - "what if you missed SIP and don’t agree with something during PR phase". For that I suggest that SIP is an iterative process - just like engineering is. The proposer would come with a proposal (SIP) with different levels of detail (it could be a few bullet points in some cases), and then he would add to it what was discussed in the PR to make it more compelling. In a way SIP is what we already have in the PRs anyways, so it’s just a little effort to organize the info. I would really encourage an effort to consolidate the info in the SIP after the PR discussion is complete.
A huge advantage of SIP is that it will be a source of technical/design documentation with historical context. This is very important for new contributors coming in and will help future refactoring, discussion across backwards compatibility issues, etc.
I understand the concerns about SIP potentially slowing down the contributions. However, I believe that true velocity comes from how quickly we can have multiple contributors pick new areas of the code and quickly expand on it. This is easier to achieve with good documentation, clean, modular code, and good unit testing that allows for quickly trying things out.
Hugo
> On Aug 2, 2017, at 7:46 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> My only concerns are
> 1. It is going to add more overhead and slow down getting a feature out. But that may not be the case if there is less overhead during the review process to offset writing up a large design, and it can be balanced with small features not needing it. 2. that I am going to miss a SIP review because I am busy and then when I complain about something wrong with a feature during a code review I will get a lot of push back because it passed SIP.
> but I can also see both sides of the coin on this. To me SIP needs to be a change to the bylaws, so if you want this change Harsha I would suggest that you write up your proposed changes to the bylaws, and we can discuss them in more detail, because it feels rather abstract to me right now. And deciding what is a large feature vs a small one also feels a bit abstract.
>
>
> - Bobby
>
>
> On Tuesday, August 1, 2017, 6:30:56 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Still +1 to introduce this only for non connectors. Maybe would want to
> skip this also for non connectors and non storm-core
> (storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
> still have small chance to need it.
>
> Despite that I voted to +1, I still worry about efforts on reviewing SIP:
> this will only work if we (in dev@ list) are open to participate and review
> SIP in desired duration. Two sides of the coin: it might incurs more active
> community, but it will just fail if community is not enough active. Each
> SIP discussion is easy to be staled if we don't care about much, and if we
> also want to introduce vote for SIP, easier to be staled.
>
> So we all should have willing to go with this. I'm OK to take additional
> load, but would like to hear others opinions as well.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
>
>> Trying to bring attention this again.
>> We currently have few big feature PRs going on and there is considerable
>> discussion about the design and Implementation etcc. My intention of
>> starting SIP is to add these details before someone goes and writes up a
>> PR and everyone has to go through reading of design and sometimes those
>> docs are not clear and we end up having long discussion on the PRs which
>> should mainly about the code review itself.
>> We should at least start making this process mandatory for any new big
>> feature especially to the storm-core. I am less concerned about
>> connectors and other parts which should have least resistive path and
>> they are usually easy to review.
>> If the devs put their thoughts and design and goes through discussion
>> and get everyone on the same line when the PR shows up it will be less
>> surprising and everyone involved know how the PR/Code supposed to work.
>>
>> -Harsha
>>
>> On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
>>> Arun,
>>> For big features we did follow design doc/review. Making it
>>> formal makes everyone to follow a process.
>>> Again this process is not for bug fixes as we stated its about New
>>> Features/Config Changes/Public interface changes. I don't think it puts
>>> any extra effort for anyone who is writing detailed JIRA but by making
>>> it formal makes everyone to add these details in a centra process. Not
>>> everyone will look at mailing list but its easier to follow a wiki page.
>>> We should atleast give it a try before we vote it out.
>>>
>>> Roshan,
>>> Adding connector should require a SIP as well and changing any
>>> public interfaces should be a KIP. Intention here is we've
>>> central place where everyone can follow in detail whats the
>>> public interface/new feature changes went in. We've changed
>>> KafkaSpout quite a bit and there is current discussion thats
>>> going to change it , having this documented in a central place
>>> will make it easy to follow and recording them in release notes
>>> as well.
>>>
>>> Taylor,
>>> We can't call it a too tedious process without even giving it a
>>> try. This has been followed to a greater success at kafka and
>>> also Flink started the process as well
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>> .
>>> If it actually proved to more of hindrance than helping the community we
>>> can move away from it.
>>>
>>> " Kafka has somewhat of a reputation for setting potentially too high a
>>> bar. I'd rather not see that happen with this community."
>>> Sure. But it also depends on the community. Just because some community
>>> enforcing too high bar that doesn't mean we are trying to do it via this
>>> process. Again we always have option if we ever veer too far in the
>>> wrong direction to bring up and improve or remove this process.
>>>
>>> We should also as a community strive to have better quality and I am
>>> hoping this will give us a chance to not only let users know what are
>>> changes coming in but also keep the dev list to have a chance and join
>>> the discussion.
>>>
>>> -Harsha
>>>
>>> On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
>>> I am for documenting and upfront design reviews, but maybe we should
>>> keep it less formal and make it part of the JIRA to start with.
>>>
>>> Do we have any upcoming features for which we would like to see a
>>> proposal? May be start with a couple of proposals
>>> and see it works out before making it formal.
>>>
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>> 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>>
>>> -0
>>>
>>> The KIP process feels kind of heavy. I'd rather start with a lighter
>>> effort like improving JIRA submissions and pull requests (some pull
>>> requests/JIRAs, even from committers and PMC members, are woefully
>>> inadequate in terms of detail), and see how that works out.
>>>
>>> I share Bobby's concern that doing so might raise the bar for
>>> contributions and potentially have a chilling effect. We don't want to
>>> scare away contributors. Kafka has somewhat of a reputation for setting
>>> potentially too high a bar. I'd rather not see that happen with this
>>> community.
>>>
>>> I will say that I like the idea of proposals for big features, ideally
>>> before any coding even begins -- so that others have a chance to
>>> collaborate. But I'm hesitant to impose too much process, voting, etc.
>>> That could scare people off.
>>>
>>> I think we should think carefully before going down this trail.
>>>
>>> -Taylor
>>>
>>> On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
>>>
>>> +1 for SIPs including a new connector. The person writing SIP can
>>> provide details about the external system for which connector is being
>>> written to let others know why a certain design decision was made. This
>>> will make it easy for reviewers.
>>>
>>> On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
>>>
>>> +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
>>> would be very useful for new contributors and others who are looking out
>>> for a feature design and decisions taken etc.
>>>
>>> Whenever a minor feature is added to a connector it may not need a
>>> separate
>>> SIP but the existing README can be updated with details for users. It
>>> can
>>> be discussed and decided apropos whether a SIP is really required for
>>> any
>>> enhancement which is not really big.
>>>
>>>
>>> On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
>>> wrote:
>>>
>>> If I am looking at the Kafka site correctly, I see that Kafka has a
>>> total
>>> of 167 KIPs so far.
>>> So I assume that minor new features would not be parrt of the SIP ?
>>>
>>> Unlike Kafka, since Storm has a number of connectors (that keep
>>> growing),
>>> I am speculating the SIP process might get somewhat unwieldy if it were
>>> to
>>> track little changes in each of the connectors.
>>>
>>> Also thinking that a SIP may not be needed to justify a new connector,
>>> but
>>> useful if we are replacing an old connector with a new one.
>>>
>>> -roshan
>>>
>>>
>>>
>>> On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
>>>
>>> Hi Bobby,
>>> In general, a KIP is required for adding New features,
>>> config
>>> changes or backward-incompatible changes. Don't require
>>> adding a KIP for bug-fixes. Devs who wants to add any
>>> features will write up a wiki which has JIRA link, mailing
>>> list discussion link and outline the Motivation, Public
>>> interface changes and protocol changes etc ..a good example
>>> here is
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 48+Delegation+token+support+for+Kafka.
>>> They can start the discussion thread once its ready and once everyone
>>> agrees its in a good shape, a Vote thread starts . Once there are
>>> required votes are in one can start the PR process and get it merged
>>> in.
>>> Each release we can collect what features/fixes especially
>>> to
>>> public interfaces that went in and roll it out in release
>>> notes. This will give a better idea for the users on what
>>> changed and added from previous version.
>>> We can only enforce this to new feature/config/backward
>>> incompatible change. Having this go through the discussion
>>> phase will give us the early feedback and potentially caught
>>> any issues before the implementation.
>>> Thanks,
>>> Harsha
>>>
>>> On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
>>>
>>> wrote:
>>>
>>> Can you please explain how KIP currently works and how you would
>>> like to see something similar in storm?
>>> If we make the process more formal we will probably have less
>>> people
>>> contributing, but we will probably have better overall patches. It
>>> is a balancing act and having never used KIP I would like to
>>> understand it better before going all in on it.
>>> - Bobby
>>>
>>>
>>> On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
>>> <ge...@gmail.com> wrote:
>>>
>>> This sounds like a good idea. KIPs seem to work well for Kafka.
>>> It's
>>> easy
>>> for discussions to get lost or just not seen on the mailing list.
>>>
>>> 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
>>>
>>> Hi All,
>>> We’ve seen good adoption of KIP approach in Kafka
>>> community
>>> and would like to see we adopt the similar approach for
>>> storm
>>> as well.
>>> Its hard to keep track of proposed changes and mailing list
>>> threads to
>>> know what all changes that are coming into and what
>>> design/backward
>>> incompatible changes being approved. It will be good to have
>>> this
>>> documented and go through discussion then Vote phase to get them
>>> approved before we merge the PRs. This will keep everyone
>>> informed of
>>> what changes happened even if they are not following the mailing
>>> list
>>> they can go to wiki to see the list of changes went into a
>>> release.
>>> Community overall will be well informed of the changes as well.
>>> Would
>>> like to hear your thoughts.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>>
>>>
>>>
>>>
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss
changes
Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
I am +1 with having a bit more structure in the process (SIP), but being cautious about limiting the overhead. I like the SIP idea, but I agree with Bobby’s point of - "what if you missed SIP and don’t agree with something during PR phase". For that I suggest that SIP is an iterative process - just like engineering is. The proposer would come with a proposal (SIP) with different levels of detail (it could be a few bullet points in some cases), and then he would add to it what was discussed in the PR to make it more compelling. In a way SIP is what we already have in the PRs anyways, so it’s just a little effort to organize the info. I would really encourage an effort to consolidate the info in the SIP after the PR discussion is complete.
A huge advantage of SIP is that it will be a source of technical/design documentation with historical context. This is very important for new contributors coming in and will help future refactoring, discussion across backwards compatibility issues, etc.
I understand the concerns about SIP potentially slowing down the contributions. However, I believe that true velocity comes from how quickly we can have multiple contributors pick new areas of the code and quickly expand on it. This is easier to achieve with good documentation, clean, modular code, and good unit testing that allows for quickly trying things out.
Hugo
> On Aug 2, 2017, at 7:46 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> My only concerns are
> 1. It is going to add more overhead and slow down getting a feature out. But that may not be the case if there is less overhead during the review process to offset writing up a large design, and it can be balanced with small features not needing it. 2. that I am going to miss a SIP review because I am busy and then when I complain about something wrong with a feature during a code review I will get a lot of push back because it passed SIP.
> but I can also see both sides of the coin on this. To me SIP needs to be a change to the bylaws, so if you want this change Harsha I would suggest that you write up your proposed changes to the bylaws, and we can discuss them in more detail, because it feels rather abstract to me right now. And deciding what is a large feature vs a small one also feels a bit abstract.
>
>
> - Bobby
>
>
> On Tuesday, August 1, 2017, 6:30:56 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Still +1 to introduce this only for non connectors. Maybe would want to
> skip this also for non connectors and non storm-core
> (storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
> still have small chance to need it.
>
> Despite that I voted to +1, I still worry about efforts on reviewing SIP:
> this will only work if we (in dev@ list) are open to participate and review
> SIP in desired duration. Two sides of the coin: it might incurs more active
> community, but it will just fail if community is not enough active. Each
> SIP discussion is easy to be staled if we don't care about much, and if we
> also want to introduce vote for SIP, easier to be staled.
>
> So we all should have willing to go with this. I'm OK to take additional
> load, but would like to hear others opinions as well.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
>
>> Trying to bring attention this again.
>> We currently have few big feature PRs going on and there is considerable
>> discussion about the design and Implementation etcc. My intention of
>> starting SIP is to add these details before someone goes and writes up a
>> PR and everyone has to go through reading of design and sometimes those
>> docs are not clear and we end up having long discussion on the PRs which
>> should mainly about the code review itself.
>> We should at least start making this process mandatory for any new big
>> feature especially to the storm-core. I am less concerned about
>> connectors and other parts which should have least resistive path and
>> they are usually easy to review.
>> If the devs put their thoughts and design and goes through discussion
>> and get everyone on the same line when the PR shows up it will be less
>> surprising and everyone involved know how the PR/Code supposed to work.
>>
>> -Harsha
>>
>> On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
>>> Arun,
>>> For big features we did follow design doc/review. Making it
>>> formal makes everyone to follow a process.
>>> Again this process is not for bug fixes as we stated its about New
>>> Features/Config Changes/Public interface changes. I don't think it puts
>>> any extra effort for anyone who is writing detailed JIRA but by making
>>> it formal makes everyone to add these details in a centra process. Not
>>> everyone will look at mailing list but its easier to follow a wiki page.
>>> We should atleast give it a try before we vote it out.
>>>
>>> Roshan,
>>> Adding connector should require a SIP as well and changing any
>>> public interfaces should be a KIP. Intention here is we've
>>> central place where everyone can follow in detail whats the
>>> public interface/new feature changes went in. We've changed
>>> KafkaSpout quite a bit and there is current discussion thats
>>> going to change it , having this documented in a central place
>>> will make it easy to follow and recording them in release notes
>>> as well.
>>>
>>> Taylor,
>>> We can't call it a too tedious process without even giving it a
>>> try. This has been followed to a greater success at kafka and
>>> also Flink started the process as well
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>> .
>>> If it actually proved to more of hindrance than helping the community we
>>> can move away from it.
>>>
>>> " Kafka has somewhat of a reputation for setting potentially too high a
>>> bar. I'd rather not see that happen with this community."
>>> Sure. But it also depends on the community. Just because some community
>>> enforcing too high bar that doesn't mean we are trying to do it via this
>>> process. Again we always have option if we ever veer too far in the
>>> wrong direction to bring up and improve or remove this process.
>>>
>>> We should also as a community strive to have better quality and I am
>>> hoping this will give us a chance to not only let users know what are
>>> changes coming in but also keep the dev list to have a chance and join
>>> the discussion.
>>>
>>> -Harsha
>>>
>>> On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
>>> I am for documenting and upfront design reviews, but maybe we should
>>> keep it less formal and make it part of the JIRA to start with.
>>>
>>> Do we have any upcoming features for which we would like to see a
>>> proposal? May be start with a couple of proposals
>>> and see it works out before making it formal.
>>>
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>> 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>>
>>> -0
>>>
>>> The KIP process feels kind of heavy. I'd rather start with a lighter
>>> effort like improving JIRA submissions and pull requests (some pull
>>> requests/JIRAs, even from committers and PMC members, are woefully
>>> inadequate in terms of detail), and see how that works out.
>>>
>>> I share Bobby's concern that doing so might raise the bar for
>>> contributions and potentially have a chilling effect. We don't want to
>>> scare away contributors. Kafka has somewhat of a reputation for setting
>>> potentially too high a bar. I'd rather not see that happen with this
>>> community.
>>>
>>> I will say that I like the idea of proposals for big features, ideally
>>> before any coding even begins -- so that others have a chance to
>>> collaborate. But I'm hesitant to impose too much process, voting, etc.
>>> That could scare people off.
>>>
>>> I think we should think carefully before going down this trail.
>>>
>>> -Taylor
>>>
>>> On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
>>>
>>> +1 for SIPs including a new connector. The person writing SIP can
>>> provide details about the external system for which connector is being
>>> written to let others know why a certain design decision was made. This
>>> will make it easy for reviewers.
>>>
>>> On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
>>>
>>> +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
>>> would be very useful for new contributors and others who are looking out
>>> for a feature design and decisions taken etc.
>>>
>>> Whenever a minor feature is added to a connector it may not need a
>>> separate
>>> SIP but the existing README can be updated with details for users. It
>>> can
>>> be discussed and decided apropos whether a SIP is really required for
>>> any
>>> enhancement which is not really big.
>>>
>>>
>>> On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
>>> wrote:
>>>
>>> If I am looking at the Kafka site correctly, I see that Kafka has a
>>> total
>>> of 167 KIPs so far.
>>> So I assume that minor new features would not be parrt of the SIP ?
>>>
>>> Unlike Kafka, since Storm has a number of connectors (that keep
>>> growing),
>>> I am speculating the SIP process might get somewhat unwieldy if it were
>>> to
>>> track little changes in each of the connectors.
>>>
>>> Also thinking that a SIP may not be needed to justify a new connector,
>>> but
>>> useful if we are replacing an old connector with a new one.
>>>
>>> -roshan
>>>
>>>
>>>
>>> On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
>>>
>>> Hi Bobby,
>>> In general, a KIP is required for adding New features,
>>> config
>>> changes or backward-incompatible changes. Don't require
>>> adding a KIP for bug-fixes. Devs who wants to add any
>>> features will write up a wiki which has JIRA link, mailing
>>> list discussion link and outline the Motivation, Public
>>> interface changes and protocol changes etc ..a good example
>>> here is
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 48+Delegation+token+support+for+Kafka.
>>> They can start the discussion thread once its ready and once everyone
>>> agrees its in a good shape, a Vote thread starts . Once there are
>>> required votes are in one can start the PR process and get it merged
>>> in.
>>> Each release we can collect what features/fixes especially
>>> to
>>> public interfaces that went in and roll it out in release
>>> notes. This will give a better idea for the users on what
>>> changed and added from previous version.
>>> We can only enforce this to new feature/config/backward
>>> incompatible change. Having this go through the discussion
>>> phase will give us the early feedback and potentially caught
>>> any issues before the implementation.
>>> Thanks,
>>> Harsha
>>>
>>> On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
>>>
>>> wrote:
>>>
>>> Can you please explain how KIP currently works and how you would
>>> like to see something similar in storm?
>>> If we make the process more formal we will probably have less
>>> people
>>> contributing, but we will probably have better overall patches. It
>>> is a balancing act and having never used KIP I would like to
>>> understand it better before going all in on it.
>>> - Bobby
>>>
>>>
>>> On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
>>> <ge...@gmail.com> wrote:
>>>
>>> This sounds like a good idea. KIPs seem to work well for Kafka.
>>> It's
>>> easy
>>> for discussions to get lost or just not seen on the mailing list.
>>>
>>> 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
>>>
>>> Hi All,
>>> We’ve seen good adoption of KIP approach in Kafka
>>> community
>>> and would like to see we adopt the similar approach for
>>> storm
>>> as well.
>>> Its hard to keep track of proposed changes and mailing list
>>> threads to
>>> know what all changes that are coming into and what
>>> design/backward
>>> incompatible changes being approved. It will be good to have
>>> this
>>> documented and go through discussion then Vote phase to get them
>>> approved before we merge the PRs. This will keep everyone
>>> informed of
>>> what changes happened even if they are not following the mailing
>>> list
>>> they can go to wiki to see the list of changes went into a
>>> release.
>>> Community overall will be well informed of the changes as well.
>>> Would
>>> like to hear your thoughts.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>>
>>>
>>>
>>>
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss
changes
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
My only concerns are
1. It is going to add more overhead and slow down getting a feature out. But that may not be the case if there is less overhead during the review process to offset writing up a large design, and it can be balanced with small features not needing it. 2. that I am going to miss a SIP review because I am busy and then when I complain about something wrong with a feature during a code review I will get a lot of push back because it passed SIP.
but I can also see both sides of the coin on this. To me SIP needs to be a change to the bylaws, so if you want this change Harsha I would suggest that you write up your proposed changes to the bylaws, and we can discuss them in more detail, because it feels rather abstract to me right now. And deciding what is a large feature vs a small one also feels a bit abstract.
- Bobby
On Tuesday, August 1, 2017, 6:30:56 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
Still +1 to introduce this only for non connectors. Maybe would want to
skip this also for non connectors and non storm-core
(storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
still have small chance to need it.
Despite that I voted to +1, I still worry about efforts on reviewing SIP:
this will only work if we (in dev@ list) are open to participate and review
SIP in desired duration. Two sides of the coin: it might incurs more active
community, but it will just fail if community is not enough active. Each
SIP discussion is easy to be staled if we don't care about much, and if we
also want to introduce vote for SIP, easier to be staled.
So we all should have willing to go with this. I'm OK to take additional
load, but would like to hear others opinions as well.
- Jungtaek Lim (HeartSaVioR)
2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
> Trying to bring attention this again.
> We currently have few big feature PRs going on and there is considerable
> discussion about the design and Implementation etcc. My intention of
> starting SIP is to add these details before someone goes and writes up a
> PR and everyone has to go through reading of design and sometimes those
> docs are not clear and we end up having long discussion on the PRs which
> should mainly about the code review itself.
> We should at least start making this process mandatory for any new big
> feature especially to the storm-core. I am less concerned about
> connectors and other parts which should have least resistive path and
> they are usually easy to review.
> If the devs put their thoughts and design and goes through discussion
> and get everyone on the same line when the PR shows up it will be less
> surprising and everyone involved know how the PR/Code supposed to work.
>
> -Harsha
>
> On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
> > Arun,
> > For big features we did follow design doc/review. Making it
> > formal makes everyone to follow a process.
> > Again this process is not for bug fixes as we stated its about New
> > Features/Config Changes/Public interface changes. I don't think it puts
> > any extra effort for anyone who is writing detailed JIRA but by making
> > it formal makes everyone to add these details in a centra process. Not
> > everyone will look at mailing list but its easier to follow a wiki page.
> > We should atleast give it a try before we vote it out.
> >
> > Roshan,
> > Adding connector should require a SIP as well and changing any
> > public interfaces should be a KIP. Intention here is we've
> > central place where everyone can follow in detail whats the
> > public interface/new feature changes went in. We've changed
> > KafkaSpout quite a bit and there is current discussion thats
> > going to change it , having this documented in a central place
> > will make it easy to follow and recording them in release notes
> > as well.
> >
> > Taylor,
> > We can't call it a too tedious process without even giving it a
> > try. This has been followed to a greater success at kafka and
> > also Flink started the process as well
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > .
> > If it actually proved to more of hindrance than helping the community we
> > can move away from it.
> >
> > " Kafka has somewhat of a reputation for setting potentially too high a
> > bar. I'd rather not see that happen with this community."
> > Sure. But it also depends on the community. Just because some community
> > enforcing too high bar that doesn't mean we are trying to do it via this
> > process. Again we always have option if we ever veer too far in the
> > wrong direction to bring up and improve or remove this process.
> >
> > We should also as a community strive to have better quality and I am
> > hoping this will give us a chance to not only let users know what are
> > changes coming in but also keep the dev list to have a chance and join
> > the discussion.
> >
> > -Harsha
> >
> > On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
> > I am for documenting and upfront design reviews, but maybe we should
> > keep it less formal and make it part of the JIRA to start with.
> >
> > Do we have any upcoming features for which we would like to see a
> > proposal? May be start with a couple of proposals
> > and see it works out before making it formal.
> >
> >
> > Thanks,
> > Arun
> >
> >
> >
> > 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
> >
> > -0
> >
> > The KIP process feels kind of heavy. I'd rather start with a lighter
> > effort like improving JIRA submissions and pull requests (some pull
> > requests/JIRAs, even from committers and PMC members, are woefully
> > inadequate in terms of detail), and see how that works out.
> >
> > I share Bobby's concern that doing so might raise the bar for
> > contributions and potentially have a chilling effect. We don't want to
> > scare away contributors. Kafka has somewhat of a reputation for setting
> > potentially too high a bar. I'd rather not see that happen with this
> > community.
> >
> > I will say that I like the idea of proposals for big features, ideally
> > before any coding even begins -- so that others have a chance to
> > collaborate. But I'm hesitant to impose too much process, voting, etc.
> > That could scare people off.
> >
> > I think we should think carefully before going down this trail.
> >
> > -Taylor
> >
> > On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> >
> > +1 for SIPs including a new connector. The person writing SIP can
> > provide details about the external system for which connector is being
> > written to let others know why a certain design decision was made. This
> > will make it easy for reviewers.
> >
> > On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
> >
> > +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
> > would be very useful for new contributors and others who are looking out
> > for a feature design and decisions taken etc.
> >
> > Whenever a minor feature is added to a connector it may not need a
> > separate
> > SIP but the existing README can be updated with details for users. It
> > can
> > be discussed and decided apropos whether a SIP is really required for
> > any
> > enhancement which is not really big.
> >
> >
> > On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
> > wrote:
> >
> > If I am looking at the Kafka site correctly, I see that Kafka has a
> > total
> > of 167 KIPs so far.
> > So I assume that minor new features would not be parrt of the SIP ?
> >
> > Unlike Kafka, since Storm has a number of connectors (that keep
> > growing),
> > I am speculating the SIP process might get somewhat unwieldy if it were
> > to
> > track little changes in each of the connectors.
> >
> > Also thinking that a SIP may not be needed to justify a new connector,
> > but
> > useful if we are replacing an old connector with a new one.
> >
> > -roshan
> >
> >
> >
> > On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
> >
> > Hi Bobby,
> > In general, a KIP is required for adding New features,
> > config
> > changes or backward-incompatible changes. Don't require
> > adding a KIP for bug-fixes. Devs who wants to add any
> > features will write up a wiki which has JIRA link, mailing
> > list discussion link and outline the Motivation, Public
> > interface changes and protocol changes etc ..a good example
> > here is
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 48+Delegation+token+support+for+Kafka.
> > They can start the discussion thread once its ready and once everyone
> > agrees its in a good shape, a Vote thread starts . Once there are
> > required votes are in one can start the PR process and get it merged
> > in.
> > Each release we can collect what features/fixes especially
> > to
> > public interfaces that went in and roll it out in release
> > notes. This will give a better idea for the users on what
> > changed and added from previous version.
> > We can only enforce this to new feature/config/backward
> > incompatible change. Having this go through the discussion
> > phase will give us the early feedback and potentially caught
> > any issues before the implementation.
> > Thanks,
> > Harsha
> >
> > On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
> >
> > wrote:
> >
> > Can you please explain how KIP currently works and how you would
> > like to see something similar in storm?
> > If we make the process more formal we will probably have less
> > people
> > contributing, but we will probably have better overall patches. It
> > is a balancing act and having never used KIP I would like to
> > understand it better before going all in on it.
> > - Bobby
> >
> >
> > On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
> > <ge...@gmail.com> wrote:
> >
> > This sounds like a good idea. KIPs seem to work well for Kafka.
> > It's
> > easy
> > for discussions to get lost or just not seen on the mailing list.
> >
> > 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
> >
> > Hi All,
> > We’ve seen good adoption of KIP approach in Kafka
> > community
> > and would like to see we adopt the similar approach for
> > storm
> > as well.
> > Its hard to keep track of proposed changes and mailing list
> > threads to
> > know what all changes that are coming into and what
> > design/backward
> > incompatible changes being approved. It will be good to have
> > this
> > documented and go through discussion then Vote phase to get them
> > approved before we merge the PRs. This will keep everyone
> > informed of
> > what changes happened even if they are not following the mailing
> > list
> > they can go to wiki to see the list of changes went into a
> > release.
> > Community overall will be well informed of the changes as well.
> > Would
> > like to hear your thoughts.
> >
> > Thanks,
> > Harsha
> >
> >
> >
> >
> >
> >
>
Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes
Posted by Jungtaek Lim <ka...@gmail.com>.
Still +1 to introduce this only for non connectors. Maybe would want to
skip this also for non connectors and non storm-core
(storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
still have small chance to need it.
Despite that I voted to +1, I still worry about efforts on reviewing SIP:
this will only work if we (in dev@ list) are open to participate and review
SIP in desired duration. Two sides of the coin: it might incurs more active
community, but it will just fail if community is not enough active. Each
SIP discussion is easy to be staled if we don't care about much, and if we
also want to introduce vote for SIP, easier to be staled.
So we all should have willing to go with this. I'm OK to take additional
load, but would like to hear others opinions as well.
- Jungtaek Lim (HeartSaVioR)
2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
> Trying to bring attention this again.
> We currently have few big feature PRs going on and there is considerable
> discussion about the design and Implementation etcc. My intention of
> starting SIP is to add these details before someone goes and writes up a
> PR and everyone has to go through reading of design and sometimes those
> docs are not clear and we end up having long discussion on the PRs which
> should mainly about the code review itself.
> We should at least start making this process mandatory for any new big
> feature especially to the storm-core. I am less concerned about
> connectors and other parts which should have least resistive path and
> they are usually easy to review.
> If the devs put their thoughts and design and goes through discussion
> and get everyone on the same line when the PR shows up it will be less
> surprising and everyone involved know how the PR/Code supposed to work.
>
> -Harsha
>
> On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
> > Arun,
> > For big features we did follow design doc/review. Making it
> > formal makes everyone to follow a process.
> > Again this process is not for bug fixes as we stated its about New
> > Features/Config Changes/Public interface changes. I don't think it puts
> > any extra effort for anyone who is writing detailed JIRA but by making
> > it formal makes everyone to add these details in a centra process. Not
> > everyone will look at mailing list but its easier to follow a wiki page.
> > We should atleast give it a try before we vote it out.
> >
> > Roshan,
> > Adding connector should require a SIP as well and changing any
> > public interfaces should be a KIP. Intention here is we've
> > central place where everyone can follow in detail whats the
> > public interface/new feature changes went in. We've changed
> > KafkaSpout quite a bit and there is current discussion thats
> > going to change it , having this documented in a central place
> > will make it easy to follow and recording them in release notes
> > as well.
> >
> > Taylor,
> > We can't call it a too tedious process without even giving it a
> > try. This has been followed to a greater success at kafka and
> > also Flink started the process as well
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > .
> > If it actually proved to more of hindrance than helping the community we
> > can move away from it.
> >
> > " Kafka has somewhat of a reputation for setting potentially too high a
> > bar. I'd rather not see that happen with this community."
> > Sure. But it also depends on the community. Just because some community
> > enforcing too high bar that doesn't mean we are trying to do it via this
> > process. Again we always have option if we ever veer too far in the
> > wrong direction to bring up and improve or remove this process.
> >
> > We should also as a community strive to have better quality and I am
> > hoping this will give us a chance to not only let users know what are
> > changes coming in but also keep the dev list to have a chance and join
> > the discussion.
> >
> > -Harsha
> >
> > On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
> > I am for documenting and upfront design reviews, but maybe we should
> > keep it less formal and make it part of the JIRA to start with.
> >
> > Do we have any upcoming features for which we would like to see a
> > proposal? May be start with a couple of proposals
> > and see it works out before making it formal.
> >
> >
> > Thanks,
> > Arun
> >
> >
> >
> > 6/9/17, 6:49 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
> >
> > -0
> >
> > The KIP process feels kind of heavy. I'd rather start with a lighter
> > effort like improving JIRA submissions and pull requests (some pull
> > requests/JIRAs, even from committers and PMC members, are woefully
> > inadequate in terms of detail), and see how that works out.
> >
> > I share Bobby's concern that doing so might raise the bar for
> > contributions and potentially have a chilling effect. We don't want to
> > scare away contributors. Kafka has somewhat of a reputation for setting
> > potentially too high a bar. I'd rather not see that happen with this
> > community.
> >
> > I will say that I like the idea of proposals for big features, ideally
> > before any coding even begins -- so that others have a chance to
> > collaborate. But I'm hesitant to impose too much process, voting, etc.
> > That could scare people off.
> >
> > I think we should think carefully before going down this trail.
> >
> > -Taylor
> >
> > On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> >
> > +1 for SIPs including a new connector. The person writing SIP can
> > provide details about the external system for which connector is being
> > written to let others know why a certain design decision was made. This
> > will make it easy for reviewers.
> >
> > On 6/9/17, 5:24 PM, "Satish Duggana" <sa...@gmail.com> wrote:
> >
> > +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
> > would be very useful for new contributors and others who are looking out
> > for a feature design and decisions taken etc.
> >
> > Whenever a minor feature is added to a connector it may not need a
> > separate
> > SIP but the existing README can be updated with details for users. It
> > can
> > be discussed and decided apropos whether a SIP is really required for
> > any
> > enhancement which is not really big.
> >
> >
> > On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ro...@hortonworks.com>
> > wrote:
> >
> > If I am looking at the Kafka site correctly, I see that Kafka has a
> > total
> > of 167 KIPs so far.
> > So I assume that minor new features would not be parrt of the SIP ?
> >
> > Unlike Kafka, since Storm has a number of connectors (that keep
> > growing),
> > I am speculating the SIP process might get somewhat unwieldy if it were
> > to
> > track little changes in each of the connectors.
> >
> > Also thinking that a SIP may not be needed to justify a new connector,
> > but
> > useful if we are replacing an old connector with a new one.
> >
> > -roshan
> >
> >
> >
> > On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
> >
> > Hi Bobby,
> > In general, a KIP is required for adding New features,
> > config
> > changes or backward-incompatible changes. Don't require
> > adding a KIP for bug-fixes. Devs who wants to add any
> > features will write up a wiki which has JIRA link, mailing
> > list discussion link and outline the Motivation, Public
> > interface changes and protocol changes etc ..a good example
> > here is
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 48+Delegation+token+support+for+Kafka.
> > They can start the discussion thread once its ready and once everyone
> > agrees its in a good shape, a Vote thread starts . Once there are
> > required votes are in one can start the PR process and get it merged
> > in.
> > Each release we can collect what features/fixes especially
> > to
> > public interfaces that went in and roll it out in release
> > notes. This will give a better idea for the users on what
> > changed and added from previous version.
> > We can only enforce this to new feature/config/backward
> > incompatible change. Having this go through the discussion
> > phase will give us the early feedback and potentially caught
> > any issues before the implementation.
> > Thanks,
> > Harsha
> >
> > On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <evans@yahoo-inc.com.invalid
> >
> > wrote:
> >
> > Can you please explain how KIP currently works and how you would
> > like to see something similar in storm?
> > If we make the process more formal we will probably have less
> > people
> > contributing, but we will probably have better overall patches. It
> > is a balancing act and having never used KIP I would like to
> > understand it better before going all in on it.
> > - Bobby
> >
> >
> > On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
> > <ge...@gmail.com> wrote:
> >
> > This sounds like a good idea. KIPs seem to work well for Kafka.
> > It's
> > easy
> > for discussions to get lost or just not seen on the mailing list.
> >
> > 2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:
> >
> > Hi All,
> > We’ve seen good adoption of KIP approach in Kafka
> > community
> > and would like to see we adopt the similar approach for
> > storm
> > as well.
> > Its hard to keep track of proposed changes and mailing list
> > threads to
> > know what all changes that are coming into and what
> > design/backward
> > incompatible changes being approved. It will be good to have
> > this
> > documented and go through discussion then Vote phase to get them
> > approved before we merge the PRs. This will keep everyone
> > informed of
> > what changes happened even if they are not following the mailing
> > list
> > they can go to wiki to see the list of changes went into a
> > release.
> > Community overall will be well informed of the changes as well.
> > Would
> > like to hear your thoughts.
> >
> > Thanks,
> > Harsha
> >
> >
> >
> >
> >
> >
>