Formalize PIP workflow

[Sijie Guo <gu...@gmail.com>]

Hi all,

We are using PIP for adopting major changes to Pulsar. It was done in a
lazy-consensus way. It was very efficient for growing the community and
driving innovations in the Pulsar community. However, we don't have a clear
workflow for PIP. It is actually causing some confusion and inconsistencies.

1) New contributors don't know where to start with new major contributions.
2) We don't have a clear definition of what are major changes.

I'd like to start a thread to discuss a few things about PIP and try to
formalize the PIP workflow.

a) What is considered a "major change" that needs a PIP?
b) What should be considered to be included in a PIP?
c) What is the process?

--

Here is my proposal.

a) Any of the following should be considered a major change:

- Any major new feature, subsystem, or piece of functionality
- Any change that impacts the public interfaces of the project.

What are the "public interfaces" of the project? All the following are
public interfaces that people build around:

- Client and Admin API.
- Wire protocol.
- Storage (both data and metadata) formats.
- User-facing scripts/command-line tools, i.e. bin/pulsar,
bin/pulsar-admin, bin/pulsar-daemon.
- Docker images
- Configuration settings
- Exposed monitoring information (Prometheus metrics, topic stats and etc).

For the most part monitoring, command-line tool changes, and configs are
added with new features so these can be done with one single PIP. And they
should be well documented.

All these major changes should be shouted out in the release notes.

b) I don't think we need to enforce a format for this PIP. However, we
should encourage people including the following format.

- Motivation: describe the problem to be solved.
- Proposed changes: describe the new thing you want to do.
- New / Changed Public Interfaces: impact on any of the "compatibility
commitments" described above. The changes should be incorporated into
documentation and called out in the release note. So everyone thinks about
them.
- Migration Plan and Compatibility: If this feature requires additional
support for a non-downtime upgrade, describe how that will work.
- Rejected Alternatives: What are the other alternatives you considered and
why are they worse?

c) What is the process?

Anyone can initiate a PIP but you shouldn't do it unless you have the
intention of getting the work done to implement.

You can write a PIP in a google doc or a gist and send an email to dev
mailing list to start a conversation with the community.

The PIP is adopted via a lazy-consensus approval process. You can start
working on the PIP unless you see concerns from other people in the
community. Try to answer those comments and improve your PIP.

Re: Formalize PIP workflow

[Guangning E <eg...@gmail.com>]

All PIP are put here, https://github.com/apache/pulsar/wiki

Thanks,
Guangning

Kun Song <so...@gmail.com> 于2020年2月3日周一 下午7:07写道：

> And https://cwiki.apache.org/confluence/spacedirectory/view.action maybe a
> good candidate.
>
> On Mon, Feb 3, 2020 at 7:01 PM Kun Song <so...@gmail.com> wrote:
>
> > There is one more thing I'm concerned about, should we find a centralized
> > place to put all PIPs? If not, PIPs will be scattered across the
> internet.
> >
> > On Mon, Feb 3, 2020 at 5:30 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> We are using PIP for adopting major changes to Pulsar. It was done in a
> >> lazy-consensus way. It was very efficient for growing the community and
> >> driving innovations in the Pulsar community. However, we don't have a
> >> clear
> >> workflow for PIP. It is actually causing some confusion and
> >> inconsistencies.
> >>
> >> 1) New contributors don't know where to start with new major
> >> contributions.
> >> 2) We don't have a clear definition of what are major changes.
> >>
> >> I'd like to start a thread to discuss a few things about PIP and try to
> >> formalize the PIP workflow.
> >>
> >> a) What is considered a "major change" that needs a PIP?
> >> b) What should be considered to be included in a PIP?
> >> c) What is the process?
> >>
> >> --
> >>
> >> Here is my proposal.
> >>
> >> a) Any of the following should be considered a major change:
> >>
> >> - Any major new feature, subsystem, or piece of functionality
> >> - Any change that impacts the public interfaces of the project.
> >>
> >> What are the "public interfaces" of the project? All the following are
> >> public interfaces that people build around:
> >>
> >> - Client and Admin API.
> >> - Wire protocol.
> >> - Storage (both data and metadata) formats.
> >> - User-facing scripts/command-line tools, i.e. bin/pulsar,
> >> bin/pulsar-admin, bin/pulsar-daemon.
> >> - Docker images
> >> - Configuration settings
> >> - Exposed monitoring information (Prometheus metrics, topic stats and
> >> etc).
> >>
> >> For the most part monitoring, command-line tool changes, and configs are
> >> added with new features so these can be done with one single PIP. And
> they
> >> should be well documented.
> >>
> >> All these major changes should be shouted out in the release notes.
> >>
> >> b) I don't think we need to enforce a format for this PIP. However, we
> >> should encourage people including the following format.
> >>
> >> - Motivation: describe the problem to be solved.
> >> - Proposed changes: describe the new thing you want to do.
> >> - New / Changed Public Interfaces: impact on any of the "compatibility
> >> commitments" described above. The changes should be incorporated into
> >> documentation and called out in the release note. So everyone thinks
> about
> >> them.
> >> - Migration Plan and Compatibility: If this feature requires additional
> >> support for a non-downtime upgrade, describe how that will work.
> >> - Rejected Alternatives: What are the other alternatives you considered
> >> and
> >> why are they worse?
> >>
> >> c) What is the process?
> >>
> >> Anyone can initiate a PIP but you shouldn't do it unless you have the
> >> intention of getting the work done to implement.
> >>
> >> You can write a PIP in a google doc or a gist and send an email to dev
> >> mailing list to start a conversation with the community.
> >>
> >> The PIP is adopted via a lazy-consensus approval process. You can start
> >> working on the PIP unless you see concerns from other people in the
> >> community. Try to answer those comments and improve your PIP.
> >>
> >
>

Re: Formalize PIP workflow

[Kun Song <so...@gmail.com>]

And https://cwiki.apache.org/confluence/spacedirectory/view.action maybe a
good candidate.

On Mon, Feb 3, 2020 at 7:01 PM Kun Song <so...@gmail.com> wrote:

> There is one more thing I'm concerned about, should we find a centralized
> place to put all PIPs? If not, PIPs will be scattered across the internet.
>
> On Mon, Feb 3, 2020 at 5:30 PM Sijie Guo <gu...@gmail.com> wrote:
>
>> Hi all,
>>
>> We are using PIP for adopting major changes to Pulsar. It was done in a
>> lazy-consensus way. It was very efficient for growing the community and
>> driving innovations in the Pulsar community. However, we don't have a
>> clear
>> workflow for PIP. It is actually causing some confusion and
>> inconsistencies.
>>
>> 1) New contributors don't know where to start with new major
>> contributions.
>> 2) We don't have a clear definition of what are major changes.
>>
>> I'd like to start a thread to discuss a few things about PIP and try to
>> formalize the PIP workflow.
>>
>> a) What is considered a "major change" that needs a PIP?
>> b) What should be considered to be included in a PIP?
>> c) What is the process?
>>
>> --
>>
>> Here is my proposal.
>>
>> a) Any of the following should be considered a major change:
>>
>> - Any major new feature, subsystem, or piece of functionality
>> - Any change that impacts the public interfaces of the project.
>>
>> What are the "public interfaces" of the project? All the following are
>> public interfaces that people build around:
>>
>> - Client and Admin API.
>> - Wire protocol.
>> - Storage (both data and metadata) formats.
>> - User-facing scripts/command-line tools, i.e. bin/pulsar,
>> bin/pulsar-admin, bin/pulsar-daemon.
>> - Docker images
>> - Configuration settings
>> - Exposed monitoring information (Prometheus metrics, topic stats and
>> etc).
>>
>> For the most part monitoring, command-line tool changes, and configs are
>> added with new features so these can be done with one single PIP. And they
>> should be well documented.
>>
>> All these major changes should be shouted out in the release notes.
>>
>> b) I don't think we need to enforce a format for this PIP. However, we
>> should encourage people including the following format.
>>
>> - Motivation: describe the problem to be solved.
>> - Proposed changes: describe the new thing you want to do.
>> - New / Changed Public Interfaces: impact on any of the "compatibility
>> commitments" described above. The changes should be incorporated into
>> documentation and called out in the release note. So everyone thinks about
>> them.
>> - Migration Plan and Compatibility: If this feature requires additional
>> support for a non-downtime upgrade, describe how that will work.
>> - Rejected Alternatives: What are the other alternatives you considered
>> and
>> why are they worse?
>>
>> c) What is the process?
>>
>> Anyone can initiate a PIP but you shouldn't do it unless you have the
>> intention of getting the work done to implement.
>>
>> You can write a PIP in a google doc or a gist and send an email to dev
>> mailing list to start a conversation with the community.
>>
>> The PIP is adopted via a lazy-consensus approval process. You can start
>> working on the PIP unless you see concerns from other people in the
>> community. Try to answer those comments and improve your PIP.
>>
>

Re: Formalize PIP workflow

[Kun Song <so...@gmail.com>]

There is one more thing I'm concerned about, should we find a centralized
place to put all PIPs? If not, PIPs will be scattered across the internet.

On Mon, Feb 3, 2020 at 5:30 PM Sijie Guo <gu...@gmail.com> wrote:

> Hi all,
>
> We are using PIP for adopting major changes to Pulsar. It was done in a
> lazy-consensus way. It was very efficient for growing the community and
> driving innovations in the Pulsar community. However, we don't have a clear
> workflow for PIP. It is actually causing some confusion and
> inconsistencies.
>
> 1) New contributors don't know where to start with new major contributions.
> 2) We don't have a clear definition of what are major changes.
>
> I'd like to start a thread to discuss a few things about PIP and try to
> formalize the PIP workflow.
>
> a) What is considered a "major change" that needs a PIP?
> b) What should be considered to be included in a PIP?
> c) What is the process?
>
> --
>
> Here is my proposal.
>
> a) Any of the following should be considered a major change:
>
> - Any major new feature, subsystem, or piece of functionality
> - Any change that impacts the public interfaces of the project.
>
> What are the "public interfaces" of the project? All the following are
> public interfaces that people build around:
>
> - Client and Admin API.
> - Wire protocol.
> - Storage (both data and metadata) formats.
> - User-facing scripts/command-line tools, i.e. bin/pulsar,
> bin/pulsar-admin, bin/pulsar-daemon.
> - Docker images
> - Configuration settings
> - Exposed monitoring information (Prometheus metrics, topic stats and etc).
>
> For the most part monitoring, command-line tool changes, and configs are
> added with new features so these can be done with one single PIP. And they
> should be well documented.
>
> All these major changes should be shouted out in the release notes.
>
> b) I don't think we need to enforce a format for this PIP. However, we
> should encourage people including the following format.
>
> - Motivation: describe the problem to be solved.
> - Proposed changes: describe the new thing you want to do.
> - New / Changed Public Interfaces: impact on any of the "compatibility
> commitments" described above. The changes should be incorporated into
> documentation and called out in the release note. So everyone thinks about
> them.
> - Migration Plan and Compatibility: If this feature requires additional
> support for a non-downtime upgrade, describe how that will work.
> - Rejected Alternatives: What are the other alternatives you considered and
> why are they worse?
>
> c) What is the process?
>
> Anyone can initiate a PIP but you shouldn't do it unless you have the
> intention of getting the work done to implement.
>
> You can write a PIP in a google doc or a gist and send an email to dev
> mailing list to start a conversation with the community.
>
> The PIP is adopted via a lazy-consensus approval process. You can start
> working on the PIP unless you see concerns from other people in the
> community. Try to answer those comments and improve your PIP.
>