[VOTE] Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

Hi All,

We recently had a discussion regarding managing Beam dependencies. Please
see [1] for the email thread and [2] for the relevant document.

This discussion resulted in following policies. I believe, these will help
keep Beam at a healthy state while allowing human intervention when needed.

(1) Human readable reports on status of Beam dependencies are generated
weekly and shared with the Beam community through the dev list.

(2) Beam components should define dependencies and their versions at the
top level.

(3) A significantly outdated dependency (identified manually or through
tooling) should result in a JIRA that is a blocker for the next release.
Release manager may choose to push the blocker to the subsequent release or
downgrade from a blocker.

(4) Dependency declarations may identify owners that are responsible for
upgrading the respective dependencies.

(5) Dependencies of Java SDK components that may cause issues to other
components if leaked should be shaded.


Please vote:
[ ] +1, Approve that we adapt these policies
[ ] -1, Do not approve (please provide specific comments)

Thanks,
Cham

[1]
https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
[2]
https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing

Re: Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

This is now live: https://beam.apache.org/contribute/dependencies/

Thanks everybody for comments and ideas.

- Cham

On Thu, Jun 21, 2018 at 5:18 PM Chamikara Jayalath <ch...@google.com>
wrote:

> https://github.com/apache/beam-site/pull/475 adds a dependencies guide to
> Website based on the discussions here and in the doc.
>
> Based on a recent discussion in the doc, I added a section on backwards
> compatibility. Basically, when upgrading dependencies, PR authors and
> reviewers should take care not to break backwards compatibility guarantee
> of Beam (till next major version) and authors of PRs that introduce
> dependency upgrades should include a statement regarding this verification
> in the form of a PR comment. I think this will be a useful addition since
> we are committed to semantic versioning.
>
> Any comments are welcome.
>
> Thanks,
> Cham
>
> On Tue, Jun 12, 2018 at 8:03 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Sounds good. I think we can add this under Technical Docs here
>> <https://beam.apache.org/contribute/>.
>>
>> Thanks,
>> Cham
>>
>> On Tue, Jun 12, 2018 at 5:35 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Makes sense. Then finding a good home on the web site is the way to go.
>>>
>>> Kenn
>>>
>>> On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <ba...@google.com>
>>> wrote:
>>>
>>>> FWIW, I also think that this has relevance for users. I am a user of
>>>> Beam not a contributor and only monitor this list at a high level. But I
>>>> think the dependency issue is something that many users have to deal with.
>>>> It has bitten us at least twice over the last few months due to the fact
>>>> that we depend on other libraries too and sometimes we get version
>>>> conflicts (which is one of the issues highlighted in the doc
>>>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit>
>>>> Cham shared). I usually go through file histories on GitHub to try to
>>>> figure out why a certain version requirement is there. It would be nice if
>>>> the reasons are maintained at a higher level easier to consume by users.
>>>>
>>>> Cheers
>>>>
>>>> -B
>>>>
>>>> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> I think this is relevant for users. It makes sense for users to know
>>>>> about how Beam work with its dependencies and understand how conflicts will
>>>>> be addressed and when dependencies will be upgraded.
>>>>>
>>>>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Do you think this has relevance for users?
>>>>>>
>>>>>> If not, it might be a good use of the new Confluence space. I'm not
>>>>>> too familiar with the way permission work, but perhaps we can have a more
>>>>>> locked down area that is for policy decisions like this.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <
>>>>>> chamikara@google.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> Based on the vote (3 PMC +1s and no -1s) and based on the
>>>>>>> discussions in the doc (seems to be mostly positive), I think we can go
>>>>>>> ahead and implement some of the policies discussed so far.
>>>>>>>
>>>>>>> I have given some of the potential action items below.
>>>>>>>
>>>>>>> * Automatically generate human readable reports on status of Beam
>>>>>>> dependencies weekly and share in dev list.
>>>>>>> * Create JIRAs for significantly outdated dependencies based on
>>>>>>> above reports.
>>>>>>> * Copy some of the component level dependency version declarations
>>>>>>> to top level.
>>>>>>> * Try to identify owners for dependencies and specify owners in
>>>>>>> comments close to dependency declarations.
>>>>>>> * Vendor any dependencies that can cause issues if leaked to other
>>>>>>> components.
>>>>>>> * Add policies discussed so far to the Web site along with reasoning
>>>>>>> (from doc).
>>>>>>>
>>>>>>> Of course, I'm happy to refine or add to these polices as needed.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 to these. Thanks for clarifying!
>>>>>>>>>
>>>>>>>>> Kenn
>>>>>>>>>
>>>>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kenn,
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +0.5
>>>>>>>>>>>
>>>>>>>>>>> I like the spirit of these policies. I think they need a little
>>>>>>>>>>> wording work. Comments inline.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Who is responsible for generating these? The mechanism or
>>>>>>>>>>> responsibility should be made clear.
>>>>>>>>>>>
>>>>>>>>>>> I clicked through a doc -> thread -> doc to find even some
>>>>>>>>>>> details. It looks like manual run of a gradle command was adopted. So the
>>>>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is described in following doc (referenced by my doc).
>>>>>>>>>>
>>>>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>>>>>>>
>>>>>>>>>> Proposal is to run an automated Jenkins job that is run weekly,
>>>>>>>>>> so no need for someone to manually generate these reports.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (2) Beam components should define dependencies and their
>>>>>>>>>>>>> versions at the top level.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> I think the big "should" works better with some guidance about
>>>>>>>>>>> when something might be an exception, or at least explicit mention that
>>>>>>>>>>> there can be rare exceptions. Unless you think that is never the case. If
>>>>>>>>>>> there are no exceptions, then say "must" and if we hit a roadblock we can
>>>>>>>>>>> revisit the policy.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>>>>>>> release or downgrade from a blocker.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> How is "significantly outdated" defined? By dev@ discussion?
>>>>>>>>>>> Seems like the right way. Anyhow that's what will happen in practice as
>>>>>>>>>>> people debate the blocker bug.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This will be either through the automated Jenkins job (see the
>>>>>>>>>> doc above, where the proposal is to flag new major versions and new minor
>>>>>>>>>> versions that are more than six months old) or manually (for any critical
>>>>>>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>>>>>>> doc). Manually identified critical dependency updates may involve a
>>>>>>>>>> discussion in the dev list.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (4) Dependency declarations may identify owners that are
>>>>>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues
>>>>>>>>>>>>> to other components if leaked should be shaded.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded"
>>>>>>>>>>> aka "vendored" packages:
>>>>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>>>>>>>
>>>>>>>>>>> With Maven, this involved a lot of boilerplate so I never did
>>>>>>>>>>> it. With Gradle, we can easily build a re-usable rule to create such a
>>>>>>>>>>> package in a couple of lines. I just opened the first WIP PR here:
>>>>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by
>>>>>>>>>>> deleting the poms anyhow so by then we should have a configuration that
>>>>>>>>>>> works to vendor our currently shaded artifacts.
>>>>>>>>>>>
>>>>>>>>>>> So I think this should be rephrased to "should be vendored" so
>>>>>>>>>>> we don't have to revise the policy.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>>>>>>
>>>>>>>>>> Here are the updated policies (and more details added to doc). I
>>>>>>>>>> agree with Ahmet's point that votes should be converted to web sites where
>>>>>>>>>> we can give more details and examples.
>>>>>>>>>>
>>>>>>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>>>>>>> community through the dev list.
>>>>>>>>>>
>>>>>>>>>> (2.a) Beam components should define dependencies and their
>>>>>>>>>> versions at the top level. There can be rare exceptions, but they should
>>>>>>>>>> come with explanations.
>>>>>>>>>>
>>>>>>>>>> (3.a) A significantly outdated dependency (identified manually
>>>>>>>>>> or through the automated Jenkins job) should result in a JIRA that is a
>>>>>>>>>> blocker for the next release. Release manager may choose to push the
>>>>>>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>>>>>>
>>>>>>>>>> (4.a) Dependency declarations may identify owners that are
>>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>>
>>>>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues
>>>>>>>>>> to other components if leaked should be vendored.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Cham
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Kenn
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Please vote:
>>>>>>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Cham
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>

Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

https://github.com/apache/beam-site/pull/475 adds a dependencies guide to
Website based on the discussions here and in the doc.

Based on a recent discussion in the doc, I added a section on backwards
compatibility. Basically, when upgrading dependencies, PR authors and
reviewers should take care not to break backwards compatibility guarantee
of Beam (till next major version) and authors of PRs that introduce
dependency upgrades should include a statement regarding this verification
in the form of a PR comment. I think this will be a useful addition since
we are committed to semantic versioning.

Any comments are welcome.

Thanks,
Cham

On Tue, Jun 12, 2018 at 8:03 AM Chamikara Jayalath <ch...@google.com>
wrote:

> Sounds good. I think we can add this under Technical Docs here
> <https://beam.apache.org/contribute/>.
>
> Thanks,
> Cham
>
> On Tue, Jun 12, 2018 at 5:35 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> Makes sense. Then finding a good home on the web site is the way to go.
>>
>> Kenn
>>
>> On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <ba...@google.com> wrote:
>>
>>> FWIW, I also think that this has relevance for users. I am a user of
>>> Beam not a contributor and only monitor this list at a high level. But I
>>> think the dependency issue is something that many users have to deal with.
>>> It has bitten us at least twice over the last few months due to the fact
>>> that we depend on other libraries too and sometimes we get version
>>> conflicts (which is one of the issues highlighted in the doc
>>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit>
>>> Cham shared). I usually go through file histories on GitHub to try to
>>> figure out why a certain version requirement is there. It would be nice if
>>> the reasons are maintained at a higher level easier to consume by users.
>>>
>>> Cheers
>>>
>>> -B
>>>
>>> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> I think this is relevant for users. It makes sense for users to know
>>>> about how Beam work with its dependencies and understand how conflicts will
>>>> be addressed and when dependencies will be upgraded.
>>>>
>>>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com>
>>>> wrote:
>>>>
>>>>> Do you think this has relevance for users?
>>>>>
>>>>> If not, it might be a good use of the new Confluence space. I'm not
>>>>> too familiar with the way permission work, but perhaps we can have a more
>>>>> locked down area that is for policy decisions like this.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions
>>>>>> in the doc (seems to be mostly positive), I think we can go ahead and
>>>>>> implement some of the policies discussed so far.
>>>>>>
>>>>>> I have given some of the potential action items below.
>>>>>>
>>>>>> * Automatically generate human readable reports on status of Beam
>>>>>> dependencies weekly and share in dev list.
>>>>>> * Create JIRAs for significantly outdated dependencies based on above
>>>>>> reports.
>>>>>> * Copy some of the component level dependency version declarations to
>>>>>> top level.
>>>>>> * Try to identify owners for dependencies and specify owners in
>>>>>> comments close to dependency declarations.
>>>>>> * Vendor any dependencies that can cause issues if leaked to other
>>>>>> components.
>>>>>> * Add policies discussed so far to the Web site along with reasoning
>>>>>> (from doc).
>>>>>>
>>>>>> Of course, I'm happy to refine or add to these polices as needed.
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 to these. Thanks for clarifying!
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Kenn,
>>>>>>>>>
>>>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +0.5
>>>>>>>>>>
>>>>>>>>>> I like the spirit of these policies. I think they need a little
>>>>>>>>>> wording work. Comments inline.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Who is responsible for generating these? The mechanism or
>>>>>>>>>> responsibility should be made clear.
>>>>>>>>>>
>>>>>>>>>> I clicked through a doc -> thread -> doc to find even some
>>>>>>>>>> details. It looks like manual run of a gradle command was adopted. So the
>>>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is described in following doc (referenced by my doc).
>>>>>>>>>
>>>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>>>>>>
>>>>>>>>> Proposal is to run an automated Jenkins job that is run weekly, so
>>>>>>>>> no need for someone to manually generate these reports.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (2) Beam components should define dependencies and their versions
>>>>>>>>>>>> at the top level.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I think the big "should" works better with some guidance about
>>>>>>>>>> when something might be an exception, or at least explicit mention that
>>>>>>>>>> there can be rare exceptions. Unless you think that is never the case. If
>>>>>>>>>> there are no exceptions, then say "must" and if we hit a roadblock we can
>>>>>>>>>> revisit the policy.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>>>>>> release or downgrade from a blocker.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> How is "significantly outdated" defined? By dev@ discussion?
>>>>>>>>>> Seems like the right way. Anyhow that's what will happen in practice as
>>>>>>>>>> people debate the blocker bug.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This will be either through the automated Jenkins job (see the doc
>>>>>>>>> above, where the proposal is to flag new major versions and new minor
>>>>>>>>> versions that are more than six months old) or manually (for any critical
>>>>>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>>>>>> doc). Manually identified critical dependency updates may involve a
>>>>>>>>> discussion in the dev list.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (4) Dependency declarations may identify owners that are
>>>>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>>>>
>>>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues
>>>>>>>>>>>> to other components if leaked should be shaded.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded"
>>>>>>>>>> aka "vendored" packages:
>>>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>>>>>>
>>>>>>>>>> With Maven, this involved a lot of boilerplate so I never did it.
>>>>>>>>>> With Gradle, we can easily build a re-usable rule to create such a package
>>>>>>>>>> in a couple of lines. I just opened the first WIP PR here:
>>>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by
>>>>>>>>>> deleting the poms anyhow so by then we should have a configuration that
>>>>>>>>>> works to vendor our currently shaded artifacts.
>>>>>>>>>>
>>>>>>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>>>>>>> don't have to revise the policy.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>>>>>
>>>>>>>>> Here are the updated policies (and more details added to doc). I
>>>>>>>>> agree with Ahmet's point that votes should be converted to web sites where
>>>>>>>>> we can give more details and examples.
>>>>>>>>>
>>>>>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>>>>>> community through the dev list.
>>>>>>>>>
>>>>>>>>> (2.a) Beam components should define dependencies and their
>>>>>>>>> versions at the top level. There can be rare exceptions, but they should
>>>>>>>>> come with explanations.
>>>>>>>>>
>>>>>>>>> (3.a) A significantly outdated dependency (identified manually or
>>>>>>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>>>>>>> blocker for the next release. Release manager may choose to push the
>>>>>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>>>>>
>>>>>>>>> (4.a) Dependency declarations may identify owners that are
>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>
>>>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues
>>>>>>>>> to other components if leaked should be vendored.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kenn
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Please vote:
>>>>>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Cham
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>>>>>>> [2]
>>>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>

Re: [VOTE] Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

Sounds good. I think we can add this under Technical Docs here
<https://beam.apache.org/contribute/>.

Thanks,
Cham

On Tue, Jun 12, 2018 at 5:35 AM Kenneth Knowles <kl...@google.com> wrote:

> Makes sense. Then finding a good home on the web site is the way to go.
>
> Kenn
>
> On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <ba...@google.com> wrote:
>
>> FWIW, I also think that this has relevance for users. I am a user of Beam
>> not a contributor and only monitor this list at a high level. But I think
>> the dependency issue is something that many users have to deal with. It has
>> bitten us at least twice over the last few months due to the fact that we
>> depend on other libraries too and sometimes we get version conflicts (which
>> is one of the issues highlighted in the doc
>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit>
>> Cham shared). I usually go through file histories on GitHub to try to
>> figure out why a certain version requirement is there. It would be nice if
>> the reasons are maintained at a higher level easier to consume by users.
>>
>> Cheers
>>
>> -B
>>
>> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote:
>>
>>> I think this is relevant for users. It makes sense for users to know
>>> about how Beam work with its dependencies and understand how conflicts will
>>> be addressed and when dependencies will be upgraded.
>>>
>>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Do you think this has relevance for users?
>>>>
>>>> If not, it might be a good use of the new Confluence space. I'm not too
>>>> familiar with the way permission work, but perhaps we can have a more
>>>> locked down area that is for policy decisions like this.
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions
>>>>> in the doc (seems to be mostly positive), I think we can go ahead and
>>>>> implement some of the policies discussed so far.
>>>>>
>>>>> I have given some of the potential action items below.
>>>>>
>>>>> * Automatically generate human readable reports on status of Beam
>>>>> dependencies weekly and share in dev list.
>>>>> * Create JIRAs for significantly outdated dependencies based on above
>>>>> reports.
>>>>> * Copy some of the component level dependency version declarations to
>>>>> top level.
>>>>> * Try to identify owners for dependencies and specify owners in
>>>>> comments close to dependency declarations.
>>>>> * Vendor any dependencies that can cause issues if leaked to other
>>>>> components.
>>>>> * Add policies discussed so far to the Web site along with reasoning
>>>>> (from doc).
>>>>>
>>>>> Of course, I'm happy to refine or add to these polices as needed.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>>
>>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 to these. Thanks for clarifying!
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>>>>> chamikara@google.com> wrote:
>>>>>>>
>>>>>>>> Hi Kenn,
>>>>>>>>
>>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +0.5
>>>>>>>>>
>>>>>>>>> I like the spirit of these policies. I think they need a little
>>>>>>>>> wording work. Comments inline.
>>>>>>>>>
>>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Who is responsible for generating these? The mechanism or
>>>>>>>>> responsibility should be made clear.
>>>>>>>>>
>>>>>>>>> I clicked through a doc -> thread -> doc to find even some
>>>>>>>>> details. It looks like manual run of a gradle command was adopted. So the
>>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is described in following doc (referenced by my doc).
>>>>>>>>
>>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>>>>>
>>>>>>>> Proposal is to run an automated Jenkins job that is run weekly, so
>>>>>>>> no need for someone to manually generate these reports.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> (2) Beam components should define dependencies and their versions
>>>>>>>>>>> at the top level.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I think the big "should" works better with some guidance about
>>>>>>>>> when something might be an exception, or at least explicit mention that
>>>>>>>>> there can be rare exceptions. Unless you think that is never the case. If
>>>>>>>>> there are no exceptions, then say "must" and if we hit a roadblock we can
>>>>>>>>> revisit the policy.
>>>>>>>>>
>>>>>>>>
>>>>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>>>>> release or downgrade from a blocker.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> How is "significantly outdated" defined? By dev@ discussion?
>>>>>>>>> Seems like the right way. Anyhow that's what will happen in practice as
>>>>>>>>> people debate the blocker bug.
>>>>>>>>>
>>>>>>>>
>>>>>>>> This will be either through the automated Jenkins job (see the doc
>>>>>>>> above, where the proposal is to flag new major versions and new minor
>>>>>>>> versions that are more than six months old) or manually (for any critical
>>>>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>>>>> doc). Manually identified critical dependency updates may involve a
>>>>>>>> discussion in the dev list.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (4) Dependency declarations may identify owners that are
>>>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>>>
>>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>>>>>>> other components if leaked should be shaded.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded"
>>>>>>>>> aka "vendored" packages:
>>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>>>>>
>>>>>>>>> With Maven, this involved a lot of boilerplate so I never did it.
>>>>>>>>> With Gradle, we can easily build a re-usable rule to create such a package
>>>>>>>>> in a couple of lines. I just opened the first WIP PR here:
>>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by
>>>>>>>>> deleting the poms anyhow so by then we should have a configuration that
>>>>>>>>> works to vendor our currently shaded artifacts.
>>>>>>>>>
>>>>>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>>>>>> don't have to revise the policy.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>>>>
>>>>>>>> Here are the updated policies (and more details added to doc). I
>>>>>>>> agree with Ahmet's point that votes should be converted to web sites where
>>>>>>>> we can give more details and examples.
>>>>>>>>
>>>>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>>>>> community through the dev list.
>>>>>>>>
>>>>>>>> (2.a) Beam components should define dependencies and their
>>>>>>>> versions at the top level. There can be rare exceptions, but they should
>>>>>>>> come with explanations.
>>>>>>>>
>>>>>>>> (3.a) A significantly outdated dependency (identified manually or
>>>>>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>>>>>> blocker for the next release. Release manager may choose to push the
>>>>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>>>>
>>>>>>>> (4.a) Dependency declarations may identify owners that are
>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>
>>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>>>>>>> other components if leaked should be vendored.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cham
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kenn
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Please vote:
>>>>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Cham
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>>>>>> [2]
>>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>

Re: [VOTE] Policies for managing Beam dependencies

[Kenneth Knowles <kl...@google.com>]

Makes sense. Then finding a good home on the web site is the way to go.

Kenn

On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <ba...@google.com> wrote:

> FWIW, I also think that this has relevance for users. I am a user of Beam
> not a contributor and only monitor this list at a high level. But I think
> the dependency issue is something that many users have to deal with. It has
> bitten us at least twice over the last few months due to the fact that we
> depend on other libraries too and sometimes we get version conflicts (which
> is one of the issues highlighted in the doc
> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit>
> Cham shared). I usually go through file histories on GitHub to try to
> figure out why a certain version requirement is there. It would be nice if
> the reasons are maintained at a higher level easier to consume by users.
>
> Cheers
>
> -B
>
> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote:
>
>> I think this is relevant for users. It makes sense for users to know
>> about how Beam work with its dependencies and understand how conflicts will
>> be addressed and when dependencies will be upgraded.
>>
>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Do you think this has relevance for users?
>>>
>>> If not, it might be a good use of the new Confluence space. I'm not too
>>> familiar with the way permission work, but perhaps we can have a more
>>> locked down area that is for policy decisions like this.
>>>
>>> Kenn
>>>
>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions
>>>> in the doc (seems to be mostly positive), I think we can go ahead and
>>>> implement some of the policies discussed so far.
>>>>
>>>> I have given some of the potential action items below.
>>>>
>>>> * Automatically generate human readable reports on status of Beam
>>>> dependencies weekly and share in dev list.
>>>> * Create JIRAs for significantly outdated dependencies based on above
>>>> reports.
>>>> * Copy some of the component level dependency version declarations to
>>>> top level.
>>>> * Try to identify owners for dependencies and specify owners in
>>>> comments close to dependency declarations.
>>>> * Vendor any dependencies that can cause issues if leaked to other
>>>> components.
>>>> * Add policies discussed so far to the Web site along with reasoning
>>>> (from doc).
>>>>
>>>> Of course, I'm happy to refine or add to these polices as needed.
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>>
>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:
>>>>>
>>>>>> +1 to these. Thanks for clarifying!
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>>>> chamikara@google.com> wrote:
>>>>>>
>>>>>>> Hi Kenn,
>>>>>>>
>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +0.5
>>>>>>>>
>>>>>>>> I like the spirit of these policies. I think they need a little
>>>>>>>> wording work. Comments inline.
>>>>>>>>
>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Who is responsible for generating these? The mechanism or
>>>>>>>> responsibility should be made clear.
>>>>>>>>
>>>>>>>> I clicked through a doc -> thread -> doc to find even some details.
>>>>>>>> It looks like manual run of a gradle command was adopted. So the
>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>>>
>>>>>>>
>>>>>>> This is described in following doc (referenced by my doc).
>>>>>>>
>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>>>>
>>>>>>> Proposal is to run an automated Jenkins job that is run weekly, so
>>>>>>> no need for someone to manually generate these reports.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> (2) Beam components should define dependencies and their versions
>>>>>>>>>> at the top level.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> I think the big "should" works better with some guidance about when
>>>>>>>> something might be an exception, or at least explicit mention that there
>>>>>>>> can be rare exceptions. Unless you think that is never the case. If there
>>>>>>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>>>>>>> the policy.
>>>>>>>>
>>>>>>>
>>>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>>>> release or downgrade from a blocker.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> How is "significantly outdated" defined? By dev@ discussion? Seems
>>>>>>>> like the right way. Anyhow that's what will happen in practice as people
>>>>>>>> debate the blocker bug.
>>>>>>>>
>>>>>>>
>>>>>>> This will be either through the automated Jenkins job (see the doc
>>>>>>> above, where the proposal is to flag new major versions and new minor
>>>>>>> versions that are more than six months old) or manually (for any critical
>>>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>>>> doc). Manually identified critical dependency updates may involve a
>>>>>>> discussion in the dev list.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> (4) Dependency declarations may identify owners that are
>>>>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>>>>
>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>>>>>> other components if leaked should be shaded.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>>>>>>> "vendored" packages:
>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>>>>
>>>>>>>> With Maven, this involved a lot of boilerplate so I never did it.
>>>>>>>> With Gradle, we can easily build a re-usable rule to create such a package
>>>>>>>> in a couple of lines. I just opened the first WIP PR here:
>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting
>>>>>>>> the poms anyhow so by then we should have a configuration that works to
>>>>>>>> vendor our currently shaded artifacts.
>>>>>>>>
>>>>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>>>>> don't have to revise the policy.
>>>>>>>>
>>>>>>>
>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>>>
>>>>>>> Here are the updated policies (and more details added to doc). I
>>>>>>> agree with Ahmet's point that votes should be converted to web sites where
>>>>>>> we can give more details and examples.
>>>>>>>
>>>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>>>> community through the dev list.
>>>>>>>
>>>>>>> (2.a) Beam components should define dependencies and their versions
>>>>>>> at the top level. There can be rare exceptions, but they should come with
>>>>>>> explanations.
>>>>>>>
>>>>>>> (3.a) A significantly outdated dependency (identified manually or
>>>>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>>>>> blocker for the next release. Release manager may choose to push the
>>>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>>>
>>>>>>> (4.a) Dependency declarations may identify owners that are
>>>>>>> responsible for upgrading the respective dependencies.
>>>>>>>
>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>>>>>> other components if leaked should be vendored.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Please vote:
>>>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Cham
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>>>>> [2]
>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>

Re: [VOTE] Policies for managing Beam dependencies

[Bashir Sadjad <ba...@google.com>]

FWIW, I also think that this has relevance for users. I am a user of Beam
not a contributor and only monitor this list at a high level. But I think
the dependency issue is something that many users have to deal with. It has
bitten us at least twice over the last few months due to the fact that we
depend on other libraries too and sometimes we get version conflicts (which
is one of the issues highlighted in the doc
<https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit>
Cham shared). I usually go through file histories on GitHub to try to
figure out why a certain version requirement is there. It would be nice if
the reasons are maintained at a higher level easier to consume by users.

Cheers

-B

On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote:

> I think this is relevant for users. It makes sense for users to know about
> how Beam work with its dependencies and understand how conflicts will be
> addressed and when dependencies will be upgraded.
>
> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>> Do you think this has relevance for users?
>>
>> If not, it might be a good use of the new Confluence space. I'm not too
>> familiar with the way permission work, but perhaps we can have a more
>> locked down area that is for policy decisions like this.
>>
>> Kenn
>>
>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions in
>>> the doc (seems to be mostly positive), I think we can go ahead and
>>> implement some of the policies discussed so far.
>>>
>>> I have given some of the potential action items below.
>>>
>>> * Automatically generate human readable reports on status of Beam
>>> dependencies weekly and share in dev list.
>>> * Create JIRAs for significantly outdated dependencies based on above
>>> reports.
>>> * Copy some of the component level dependency version declarations to
>>> top level.
>>> * Try to identify owners for dependencies and specify owners in comments
>>> close to dependency declarations.
>>> * Vendor any dependencies that can cause issues if leaked to other
>>> components.
>>> * Add policies discussed so far to the Web site along with reasoning
>>> (from doc).
>>>
>>> Of course, I'm happy to refine or add to these polices as needed.
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> +1 to these. Thanks for clarifying!
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>
>>>>>> Hi Kenn,
>>>>>>
>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +0.5
>>>>>>>
>>>>>>> I like the spirit of these policies. I think they need a little
>>>>>>> wording work. Comments inline.
>>>>>>>
>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>>
>>>>>>>>
>>>>>>> Who is responsible for generating these? The mechanism or
>>>>>>> responsibility should be made clear.
>>>>>>>
>>>>>>> I clicked through a doc -> thread -> doc to find even some details.
>>>>>>> It looks like manual run of a gradle command was adopted. So the
>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>>
>>>>>>
>>>>>> This is described in following doc (referenced by my doc).
>>>>>>
>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>>>
>>>>>> Proposal is to run an automated Jenkins job that is run weekly, so no
>>>>>> need for someone to manually generate these reports.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> (2) Beam components should define dependencies and their versions at
>>>>>>>>> the top level.
>>>>>>>>>
>>>>>>>>
>>>>>>> I think the big "should" works better with some guidance about when
>>>>>>> something might be an exception, or at least explicit mention that there
>>>>>>> can be rare exceptions. Unless you think that is never the case. If there
>>>>>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>>>>>> the policy.
>>>>>>>
>>>>>>
>>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>>> release or downgrade from a blocker.
>>>>>>>>>
>>>>>>>>
>>>>>>> How is "significantly outdated" defined? By dev@ discussion? Seems
>>>>>>> like the right way. Anyhow that's what will happen in practice as people
>>>>>>> debate the blocker bug.
>>>>>>>
>>>>>>
>>>>>> This will be either through the automated Jenkins job (see the doc
>>>>>> above, where the proposal is to flag new major versions and new minor
>>>>>> versions that are more than six months old) or manually (for any critical
>>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>>> doc). Manually identified critical dependency updates may involve a
>>>>>> discussion in the dev list.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> (4) Dependency declarations may identify owners that are responsible
>>>>>>>>> for upgrading the respective dependencies.
>>>>>>>>>
>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>>>>> other components if leaked should be shaded.
>>>>>>>>>
>>>>>>>>
>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>>>>>> "vendored" packages:
>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>>>
>>>>>>> With Maven, this involved a lot of boilerplate so I never did it.
>>>>>>> With Gradle, we can easily build a re-usable rule to create such a package
>>>>>>> in a couple of lines. I just opened the first WIP PR here:
>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting
>>>>>>> the poms anyhow so by then we should have a configuration that works to
>>>>>>> vendor our currently shaded artifacts.
>>>>>>>
>>>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>>>> don't have to revise the policy.
>>>>>>>
>>>>>>
>>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>>
>>>>>> Here are the updated policies (and more details added to doc). I
>>>>>> agree with Ahmet's point that votes should be converted to web sites where
>>>>>> we can give more details and examples.
>>>>>>
>>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>>> community through the dev list.
>>>>>>
>>>>>> (2.a) Beam components should define dependencies and their versions
>>>>>> at the top level. There can be rare exceptions, but they should come with
>>>>>> explanations.
>>>>>>
>>>>>> (3.a) A significantly outdated dependency (identified manually or
>>>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>>>> blocker for the next release. Release manager may choose to push the
>>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>>
>>>>>> (4.a) Dependency declarations may identify owners that are
>>>>>> responsible for upgrading the respective dependencies.
>>>>>>
>>>>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>>>>> other components if leaked should be vendored.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Please vote:
>>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>>>> [2]
>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>

Re: [VOTE] Policies for managing Beam dependencies

[Ahmet Altay <al...@google.com>]

I think this is relevant for users. It makes sense for users to know about
how Beam work with its dependencies and understand how conflicts will be
addressed and when dependencies will be upgraded.

On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <kl...@google.com> wrote:

> Do you think this has relevance for users?
>
> If not, it might be a good use of the new Confluence space. I'm not too
> familiar with the way permission work, but perhaps we can have a more
> locked down area that is for policy decisions like this.
>
> Kenn
>
> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi All,
>>
>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions in
>> the doc (seems to be mostly positive), I think we can go ahead and
>> implement some of the policies discussed so far.
>>
>> I have given some of the potential action items below.
>>
>> * Automatically generate human readable reports on status of Beam
>> dependencies weekly and share in dev list.
>> * Create JIRAs for significantly outdated dependencies based on above
>> reports.
>> * Copy some of the component level dependency version declarations to top
>> level.
>> * Try to identify owners for dependencies and specify owners in comments
>> close to dependency declarations.
>> * Vendor any dependencies that can cause issues if leaked to other
>> components.
>> * Add policies discussed so far to the Web site along with reasoning
>> (from doc).
>>
>> Of course, I'm happy to refine or add to these polices as needed.
>>
>> Thanks,
>> Cham
>>
>>
>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> +1 to these. Thanks for clarifying!
>>>>
>>>> Kenn
>>>>
>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> Hi Kenn,
>>>>>
>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>>
>>>>>> +0.5
>>>>>>
>>>>>> I like the spirit of these policies. I think they need a little
>>>>>> wording work. Comments inline.
>>>>>>
>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>>> chamikara@google.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>>
>>>>>>>
>>>>>> Who is responsible for generating these? The mechanism or
>>>>>> responsibility should be made clear.
>>>>>>
>>>>>> I clicked through a doc -> thread -> doc to find even some details.
>>>>>> It looks like manual run of a gradle command was adopted. So the
>>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>>
>>>>>
>>>>> This is described in following doc (referenced by my doc).
>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRX
>>>>> sO72BpBwA/edit#
>>>>>
>>>>> Proposal is to run an automated Jenkins job that is run weekly, so no
>>>>> need for someone to manually generate these reports.
>>>>>
>>>>>
>>>>>>
>>>>>> (2) Beam components should define dependencies and their versions at
>>>>>>>> the top level.
>>>>>>>>
>>>>>>>
>>>>>> I think the big "should" works better with some guidance about when
>>>>>> something might be an exception, or at least explicit mention that there
>>>>>> can be rare exceptions. Unless you think that is never the case. If there
>>>>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>>>>> the policy.
>>>>>>
>>>>>
>>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>>> release or downgrade from a blocker.
>>>>>>>>
>>>>>>>
>>>>>> How is "significantly outdated" defined? By dev@ discussion? Seems
>>>>>> like the right way. Anyhow that's what will happen in practice as people
>>>>>> debate the blocker bug.
>>>>>>
>>>>>
>>>>> This will be either through the automated Jenkins job (see the doc
>>>>> above, where the proposal is to flag new major versions and new minor
>>>>> versions that are more than six months old) or manually (for any critical
>>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>>> doc). Manually identified critical dependency updates may involve a
>>>>> discussion in the dev list.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> (4) Dependency declarations may identify owners that are responsible
>>>>>>>> for upgrading the respective dependencies.
>>>>>>>>
>>>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>>>> other components if leaked should be shaded.
>>>>>>>>
>>>>>>>
>>>>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>>>>> "vendored" packages: https://lists.apache.org/thread.html/
>>>>>> 12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%
>>>>>> 3Cdev.beam.apache.org%3E
>>>>>>
>>>>>> With Maven, this involved a lot of boilerplate so I never did it.
>>>>>> With Gradle, we can easily build a re-usable rule to create such a package
>>>>>> in a couple of lines. I just opened the first WIP PR here:
>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting
>>>>>> the poms anyhow so by then we should have a configuration that works to
>>>>>> vendor our currently shaded artifacts.
>>>>>>
>>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>>> don't have to revise the policy.
>>>>>>
>>>>>
>>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>>
>>>>> Here are the updated policies (and more details added to doc). I agree
>>>>> with Ahmet's point that votes should be converted to web sites where we can
>>>>> give more details and examples.
>>>>>
>>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>>> community through the dev list.
>>>>>
>>>>> (2.a) Beam components should define dependencies and their versions
>>>>> at the top level. There can be rare exceptions, but they should come with
>>>>> explanations.
>>>>>
>>>>> (3.a) A significantly outdated dependency (identified manually or
>>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>>> blocker for the next release. Release manager may choose to push the
>>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>>
>>>>> (4.a) Dependency declarations may identify owners that are
>>>>> responsible for upgrading the respective dependencies.
>>>>>
>>>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>>>> other components if leaked should be vendored.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>>
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Please vote:
>>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cham
>>>>>>>>
>>>>>>>> [1] https://lists.apache.org/thread.html/
>>>>>>>> 8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%
>>>>>>>> 3Cdev.beam.apache.org%3E
>>>>>>>> [2] https://docs.google.com/document/d/15m1MziZ5TNd9rh_
>>>>>>>> XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>>
>>>>>>>
>>>>>>>

Re: [VOTE] Policies for managing Beam dependencies

[Kenneth Knowles <kl...@google.com>]

Do you think this has relevance for users?

If not, it might be a good use of the new Confluence space. I'm not too
familiar with the way permission work, but perhaps we can have a more
locked down area that is for policy decisions like this.

Kenn

On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath <ch...@google.com>
wrote:

> Hi All,
>
> Based on the vote (3 PMC +1s and no -1s) and based on the discussions in
> the doc (seems to be mostly positive), I think we can go ahead and
> implement some of the policies discussed so far.
>
> I have given some of the potential action items below.
>
> * Automatically generate human readable reports on status of Beam
> dependencies weekly and share in dev list.
> * Create JIRAs for significantly outdated dependencies based on above
> reports.
> * Copy some of the component level dependency version declarations to top
> level.
> * Try to identify owners for dependencies and specify owners in comments
> close to dependency declarations.
> * Vendor any dependencies that can cause issues if leaked to other
> components.
> * Add policies discussed so far to the Web site along with reasoning (from
> doc).
>
> Of course, I'm happy to refine or add to these polices as needed.
>
> Thanks,
> Cham
>
>
> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> +1
>>
>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> +1 to these. Thanks for clarifying!
>>>
>>> Kenn
>>>
>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>> Hi Kenn,
>>>>
>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> +0.5
>>>>>
>>>>> I like the spirit of these policies. I think they need a little
>>>>> wording work. Comments inline.
>>>>>
>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>>> chamikara@google.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>>
>>>>>>
>>>>> Who is responsible for generating these? The mechanism or
>>>>> responsibility should be made clear.
>>>>>
>>>>> I clicked through a doc -> thread -> doc to find even some details. It
>>>>> looks like manual run of a gradle command was adopted. So the
>>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>>> and feel free to complain or do it yourself if you don't see it"
>>>>>
>>>>
>>>> This is described in following doc (referenced by my doc).
>>>>
>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>>
>>>> Proposal is to run an automated Jenkins job that is run weekly, so no
>>>> need for someone to manually generate these reports.
>>>>
>>>>
>>>>>
>>>>> (2) Beam components should define dependencies and their versions at
>>>>>>> the top level.
>>>>>>>
>>>>>>
>>>>> I think the big "should" works better with some guidance about when
>>>>> something might be an exception, or at least explicit mention that there
>>>>> can be rare exceptions. Unless you think that is never the case. If there
>>>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>>>> the policy.
>>>>>
>>>>
>>>> The idea was to allow exceptions. Added more details to the doc.
>>>>
>>>>
>>>>>
>>>>>
>>>>> (3) A significantly outdated dependency (identified manually or
>>>>>>> through tooling) should result in a JIRA that is a blocker for the next
>>>>>>> release. Release manager may choose to push the blocker to the subsequent
>>>>>>> release or downgrade from a blocker.
>>>>>>>
>>>>>>
>>>>> How is "significantly outdated" defined? By dev@ discussion? Seems
>>>>> like the right way. Anyhow that's what will happen in practice as people
>>>>> debate the blocker bug.
>>>>>
>>>>
>>>> This will be either through the automated Jenkins job (see the doc
>>>> above, where the proposal is to flag new major versions and new minor
>>>> versions that are more than six months old) or manually (for any critical
>>>> updates that will not be captured by the Jenkins job) (more details in the
>>>> doc). Manually identified critical dependency updates may involve a
>>>> discussion in the dev list.
>>>>
>>>>
>>>>>
>>>>>
>>>>> (4) Dependency declarations may identify owners that are responsible
>>>>>>> for upgrading the respective dependencies.
>>>>>>>
>>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>>> other components if leaked should be shaded.
>>>>>>>
>>>>>>
>>>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>>>> "vendored" packages:
>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>>
>>>>> With Maven, this involved a lot of boilerplate so I never did it. With
>>>>> Gradle, we can easily build a re-usable rule to create such a package in a
>>>>> couple of lines. I just opened the first WIP PR here:
>>>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting
>>>>> the poms anyhow so by then we should have a configuration that works to
>>>>> vendor our currently shaded artifacts.
>>>>>
>>>>> So I think this should be rephrased to "should be vendored" so we
>>>>> don't have to revise the policy.
>>>>>
>>>>
>>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>>
>>>> Here are the updated policies (and more details added to doc). I agree
>>>> with Ahmet's point that votes should be converted to web sites where we can
>>>> give more details and examples.
>>>>
>>>> (1.a) Human readable reports on status of Beam dependencies are
>>>> generated weekly by an automated Jenkins job and shared with the Beam
>>>> community through the dev list.
>>>>
>>>> (2.a) Beam components should define dependencies and their versions at
>>>> the top level. There can be rare exceptions, but they should come with
>>>> explanations.
>>>>
>>>> (3.a) A significantly outdated dependency (identified manually or
>>>> through the automated Jenkins job) should result in a JIRA that is a
>>>> blocker for the next release. Release manager may choose to push the
>>>> blocker to the subsequent release or downgrade from a blocker.
>>>>
>>>> (4.a) Dependency declarations may identify owners that are responsible
>>>> for upgrading the respective dependencies.
>>>>
>>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>>> other components if leaked should be vendored.
>>>>
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>>
>>>>>
>>>>> Kenn
>>>>>
>>>>>
>>>>>
>>>>>> Please vote:
>>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> [1]
>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>>> [2]
>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>>
>>>>>>
>>>>>>

Re: [VOTE] Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

Hi All,

Based on the vote (3 PMC +1s and no -1s) and based on the discussions in
the doc (seems to be mostly positive), I think we can go ahead and
implement some of the policies discussed so far.

I have given some of the potential action items below.

* Automatically generate human readable reports on status of Beam
dependencies weekly and share in dev list.
* Create JIRAs for significantly outdated dependencies based on above
reports.
* Copy some of the component level dependency version declarations to top
level.
* Try to identify owners for dependencies and specify owners in comments
close to dependency declarations.
* Vendor any dependencies that can cause issues if leaked to other
components.
* Add policies discussed so far to the Web site along with reasoning (from
doc).

Of course, I'm happy to refine or add to these polices as needed.

Thanks,
Cham


On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote:

> +1
>
> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> +1 to these. Thanks for clarifying!
>>
>> Kenn
>>
>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Hi Kenn,
>>>
>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> +0.5
>>>>
>>>> I like the spirit of these policies. I think they need a little wording
>>>> work. Comments inline.
>>>>
>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>>
>>>>>>
>>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>>
>>>>>
>>>> Who is responsible for generating these? The mechanism or
>>>> responsibility should be made clear.
>>>>
>>>> I clicked through a doc -> thread -> doc to find even some details. It
>>>> looks like manual run of a gradle command was adopted. So the
>>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>>> and feel free to complain or do it yourself if you don't see it"
>>>>
>>>
>>> This is described in following doc (referenced by my doc).
>>>
>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>>
>>> Proposal is to run an automated Jenkins job that is run weekly, so no
>>> need for someone to manually generate these reports.
>>>
>>>
>>>>
>>>> (2) Beam components should define dependencies and their versions at
>>>>>> the top level.
>>>>>>
>>>>>
>>>> I think the big "should" works better with some guidance about when
>>>> something might be an exception, or at least explicit mention that there
>>>> can be rare exceptions. Unless you think that is never the case. If there
>>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>>> the policy.
>>>>
>>>
>>> The idea was to allow exceptions. Added more details to the doc.
>>>
>>>
>>>>
>>>>
>>>> (3) A significantly outdated dependency (identified manually or through
>>>>>> tooling) should result in a JIRA that is a blocker for the next release.
>>>>>> Release manager may choose to push the blocker to the subsequent release or
>>>>>> downgrade from a blocker.
>>>>>>
>>>>>
>>>> How is "significantly outdated" defined? By dev@ discussion? Seems
>>>> like the right way. Anyhow that's what will happen in practice as people
>>>> debate the blocker bug.
>>>>
>>>
>>> This will be either through the automated Jenkins job (see the doc
>>> above, where the proposal is to flag new major versions and new minor
>>> versions that are more than six months old) or manually (for any critical
>>> updates that will not be captured by the Jenkins job) (more details in the
>>> doc). Manually identified critical dependency updates may involve a
>>> discussion in the dev list.
>>>
>>>
>>>>
>>>>
>>>> (4) Dependency declarations may identify owners that are responsible
>>>>>> for upgrading the respective dependencies.
>>>>>>
>>>>>> (5) Dependencies of Java SDK components that may cause issues to
>>>>>> other components if leaked should be shaded.
>>>>>>
>>>>>
>>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>>> "vendored" packages:
>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>>
>>>> With Maven, this involved a lot of boilerplate so I never did it. With
>>>> Gradle, we can easily build a re-usable rule to create such a package in a
>>>> couple of lines. I just opened the first WIP PR here:
>>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting the
>>>> poms anyhow so by then we should have a configuration that works to vendor
>>>> our currently shaded artifacts.
>>>>
>>>> So I think this should be rephrased to "should be vendored" so we don't
>>>> have to revise the policy.
>>>>
>>>
>>> Thanks for the pointer. I agree that vendoring is a good approach.
>>>
>>> Here are the updated policies (and more details added to doc). I agree
>>> with Ahmet's point that votes should be converted to web sites where we can
>>> give more details and examples.
>>>
>>> (1.a) Human readable reports on status of Beam dependencies are
>>> generated weekly by an automated Jenkins job and shared with the Beam
>>> community through the dev list.
>>>
>>> (2.a) Beam components should define dependencies and their versions at
>>> the top level. There can be rare exceptions, but they should come with
>>> explanations.
>>>
>>> (3.a) A significantly outdated dependency (identified manually or
>>> through the automated Jenkins job) should result in a JIRA that is a
>>> blocker for the next release. Release manager may choose to push the
>>> blocker to the subsequent release or downgrade from a blocker.
>>>
>>> (4.a) Dependency declarations may identify owners that are responsible
>>> for upgrading the respective dependencies.
>>>
>>> (5.a) Dependencies of Java SDK components that may cause issues to
>>> other components if leaked should be vendored.
>>>
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>>>
>>>> Kenn
>>>>
>>>>
>>>>
>>>>> Please vote:
>>>>>> [ ] +1, Approve that we adapt these policies
>>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>> [1]
>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>>> [2]
>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>>
>>>>>
>>>>>

Re: [VOTE] Policies for managing Beam dependencies

[Lukasz Cwik <lc...@google.com>]

+1

On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <kl...@google.com> wrote:

> +1 to these. Thanks for clarifying!
>
> Kenn
>
> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi Kenn,
>>
>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> +0.5
>>>
>>> I like the spirit of these policies. I think they need a little wording
>>> work. Comments inline.
>>>
>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <chamikara@google.com
>>>> > wrote:
>>>>>
>>>>>
>>>>> (1) Human readable reports on status of Beam dependencies are
>>>>> generated weekly and shared with the Beam community through the dev list.
>>>>>
>>>>
>>> Who is responsible for generating these? The mechanism or responsibility
>>> should be made clear.
>>>
>>> I clicked through a doc -> thread -> doc to find even some details. It
>>> looks like manual run of a gradle command was adopted. So the
>>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>>> and feel free to complain or do it yourself if you don't see it"
>>>
>>
>> This is described in following doc (referenced by my doc).
>>
>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>>
>> Proposal is to run an automated Jenkins job that is run weekly, so no
>> need for someone to manually generate these reports.
>>
>>
>>>
>>> (2) Beam components should define dependencies and their versions at the
>>>>> top level.
>>>>>
>>>>
>>> I think the big "should" works better with some guidance about when
>>> something might be an exception, or at least explicit mention that there
>>> can be rare exceptions. Unless you think that is never the case. If there
>>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>>> the policy.
>>>
>>
>> The idea was to allow exceptions. Added more details to the doc.
>>
>>
>>>
>>>
>>> (3) A significantly outdated dependency (identified manually or through
>>>>> tooling) should result in a JIRA that is a blocker for the next release.
>>>>> Release manager may choose to push the blocker to the subsequent release or
>>>>> downgrade from a blocker.
>>>>>
>>>>
>>> How is "significantly outdated" defined? By dev@ discussion? Seems like
>>> the right way. Anyhow that's what will happen in practice as people debate
>>> the blocker bug.
>>>
>>
>> This will be either through the automated Jenkins job (see the doc above,
>> where the proposal is to flag new major versions and new minor versions
>> that are more than six months old) or manually (for any critical updates
>> that will not be captured by the Jenkins job) (more details in the doc).
>> Manually identified critical dependency updates may involve a discussion in
>> the dev list.
>>
>>
>>>
>>>
>>> (4) Dependency declarations may identify owners that are responsible for
>>>>> upgrading the respective dependencies.
>>>>>
>>>>> (5) Dependencies of Java SDK components that may cause issues to other
>>>>> components if leaked should be shaded.
>>>>>
>>>>
>>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>>> "vendored" packages:
>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>>
>>> With Maven, this involved a lot of boilerplate so I never did it. With
>>> Gradle, we can easily build a re-usable rule to create such a package in a
>>> couple of lines. I just opened the first WIP PR here:
>>> https://github.com/apache/beam/pull/5570 it is blocked by deleting the
>>> poms anyhow so by then we should have a configuration that works to vendor
>>> our currently shaded artifacts.
>>>
>>> So I think this should be rephrased to "should be vendored" so we don't
>>> have to revise the policy.
>>>
>>
>> Thanks for the pointer. I agree that vendoring is a good approach.
>>
>> Here are the updated policies (and more details added to doc). I agree
>> with Ahmet's point that votes should be converted to web sites where we can
>> give more details and examples.
>>
>> (1.a) Human readable reports on status of Beam dependencies are generated
>> weekly by an automated Jenkins job and shared with the Beam community
>> through the dev list.
>>
>> (2.a) Beam components should define dependencies and their versions at
>> the top level. There can be rare exceptions, but they should come with
>> explanations.
>>
>> (3.a) A significantly outdated dependency (identified manually or
>> through the automated Jenkins job) should result in a JIRA that is a
>> blocker for the next release. Release manager may choose to push the
>> blocker to the subsequent release or downgrade from a blocker.
>>
>> (4.a) Dependency declarations may identify owners that are responsible
>> for upgrading the respective dependencies.
>>
>> (5.a) Dependencies of Java SDK components that may cause issues to other
>> components if leaked should be vendored.
>>
>>
>> Thanks,
>> Cham
>>
>>
>>>
>>> Kenn
>>>
>>>
>>>
>>>> Please vote:
>>>>> [ ] +1, Approve that we adapt these policies
>>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> [1]
>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>>> [2]
>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>>
>>>>
>>>>

Re: [VOTE] Policies for managing Beam dependencies

[Kenneth Knowles <kl...@google.com>]

+1 to these. Thanks for clarifying!

Kenn

On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath <ch...@google.com>
wrote:

> Hi Kenn,
>
> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:
>
>> +0.5
>>
>> I like the spirit of these policies. I think they need a little wording
>> work. Comments inline.
>>
>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>>
>>>>
>>>> (1) Human readable reports on status of Beam dependencies are generated
>>>> weekly and shared with the Beam community through the dev list.
>>>>
>>>
>> Who is responsible for generating these? The mechanism or responsibility
>> should be made clear.
>>
>> I clicked through a doc -> thread -> doc to find even some details. It
>> looks like manual run of a gradle command was adopted. So the
>> responsibility needs an owner, even if it is "unspecified volunteer on dev@
>> and feel free to complain or do it yourself if you don't see it"
>>
>
> This is described in following doc (referenced by my doc).
>
> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#
>
> Proposal is to run an automated Jenkins job that is run weekly, so no need
> for someone to manually generate these reports.
>
>
>>
>> (2) Beam components should define dependencies and their versions at the
>>>> top level.
>>>>
>>>
>> I think the big "should" works better with some guidance about when
>> something might be an exception, or at least explicit mention that there
>> can be rare exceptions. Unless you think that is never the case. If there
>> are no exceptions, then say "must" and if we hit a roadblock we can revisit
>> the policy.
>>
>
> The idea was to allow exceptions. Added more details to the doc.
>
>
>>
>>
>> (3) A significantly outdated dependency (identified manually or through
>>>> tooling) should result in a JIRA that is a blocker for the next release.
>>>> Release manager may choose to push the blocker to the subsequent release or
>>>> downgrade from a blocker.
>>>>
>>>
>> How is "significantly outdated" defined? By dev@ discussion? Seems like
>> the right way. Anyhow that's what will happen in practice as people debate
>> the blocker bug.
>>
>
> This will be either through the automated Jenkins job (see the doc above,
> where the proposal is to flag new major versions and new minor versions
> that are more than six months old) or manually (for any critical updates
> that will not be captured by the Jenkins job) (more details in the doc).
> Manually identified critical dependency updates may involve a discussion in
> the dev list.
>
>
>>
>>
>> (4) Dependency declarations may identify owners that are responsible for
>>>> upgrading the respective dependencies.
>>>>
>>>> (5) Dependencies of Java SDK components that may cause issues to other
>>>> components if leaked should be shaded.
>>>>
>>>
>> We previously agreed upon our intent to migrate to "pre-shaded" aka
>> "vendored" packages:
>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>>
>> With Maven, this involved a lot of boilerplate so I never did it. With
>> Gradle, we can easily build a re-usable rule to create such a package in a
>> couple of lines. I just opened the first WIP PR here:
>> https://github.com/apache/beam/pull/5570 it is blocked by deleting the
>> poms anyhow so by then we should have a configuration that works to vendor
>> our currently shaded artifacts.
>>
>> So I think this should be rephrased to "should be vendored" so we don't
>> have to revise the policy.
>>
>
> Thanks for the pointer. I agree that vendoring is a good approach.
>
> Here are the updated policies (and more details added to doc). I agree
> with Ahmet's point that votes should be converted to web sites where we can
> give more details and examples.
>
> (1.a) Human readable reports on status of Beam dependencies are generated
> weekly by an automated Jenkins job and shared with the Beam community
> through the dev list.
>
> (2.a) Beam components should define dependencies and their versions at
> the top level. There can be rare exceptions, but they should come with
> explanations.
>
> (3.a) A significantly outdated dependency (identified manually or through
> the automated Jenkins job) should result in a JIRA that is a blocker for
> the next release. Release manager may choose to push the blocker to the
> subsequent release or downgrade from a blocker.
>
> (4.a) Dependency declarations may identify owners that are responsible
> for upgrading the respective dependencies.
>
> (5.a) Dependencies of Java SDK components that may cause issues to other
> components if leaked should be vendored.
>
>
> Thanks,
> Cham
>
>
>>
>> Kenn
>>
>>
>>
>>> Please vote:
>>>> [ ] +1, Approve that we adapt these policies
>>>> [ ] -1, Do not approve (please provide specific comments)
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>>> [2]
>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>>
>>>
>>>

Re: [VOTE] Policies for managing Beam dependencies

[Chamikara Jayalath <ch...@google.com>]

Hi Kenn,

On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <kl...@google.com> wrote:

> +0.5
>
> I like the spirit of these policies. I think they need a little wording
> work. Comments inline.
>
> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <ch...@google.com>
>> wrote:
>>>
>>>
>>> (1) Human readable reports on status of Beam dependencies are generated
>>> weekly and shared with the Beam community through the dev list.
>>>
>>
> Who is responsible for generating these? The mechanism or responsibility
> should be made clear.
>
> I clicked through a doc -> thread -> doc to find even some details. It
> looks like manual run of a gradle command was adopted. So the
> responsibility needs an owner, even if it is "unspecified volunteer on dev@
> and feel free to complain or do it yourself if you don't see it"
>

This is described in following doc (referenced by my doc).
https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#

Proposal is to run an automated Jenkins job that is run weekly, so no need
for someone to manually generate these reports.


>
> (2) Beam components should define dependencies and their versions at the
>>> top level.
>>>
>>
> I think the big "should" works better with some guidance about when
> something might be an exception, or at least explicit mention that there
> can be rare exceptions. Unless you think that is never the case. If there
> are no exceptions, then say "must" and if we hit a roadblock we can revisit
> the policy.
>

The idea was to allow exceptions. Added more details to the doc.


>
>
> (3) A significantly outdated dependency (identified manually or through
>>> tooling) should result in a JIRA that is a blocker for the next release.
>>> Release manager may choose to push the blocker to the subsequent release or
>>> downgrade from a blocker.
>>>
>>
> How is "significantly outdated" defined? By dev@ discussion? Seems like
> the right way. Anyhow that's what will happen in practice as people debate
> the blocker bug.
>

This will be either through the automated Jenkins job (see the doc above,
where the proposal is to flag new major versions and new minor versions
that are more than six months old) or manually (for any critical updates
that will not be captured by the Jenkins job) (more details in the doc).
Manually identified critical dependency updates may involve a discussion in
the dev list.


>
>
> (4) Dependency declarations may identify owners that are responsible for
>>> upgrading the respective dependencies.
>>>
>>> (5) Dependencies of Java SDK components that may cause issues to other
>>> components if leaked should be shaded.
>>>
>>
> We previously agreed upon our intent to migrate to "pre-shaded" aka
> "vendored" packages:
> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E
>
> With Maven, this involved a lot of boilerplate so I never did it. With
> Gradle, we can easily build a re-usable rule to create such a package in a
> couple of lines. I just opened the first WIP PR here:
> https://github.com/apache/beam/pull/5570 it is blocked by deleting the
> poms anyhow so by then we should have a configuration that works to vendor
> our currently shaded artifacts.
>
> So I think this should be rephrased to "should be vendored" so we don't
> have to revise the policy.
>

Thanks for the pointer. I agree that vendoring is a good approach.

Here are the updated policies (and more details added to doc). I agree with
Ahmet's point that votes should be converted to web sites where we can give
more details and examples.

(1.a) Human readable reports on status of Beam dependencies are generated
weekly by an automated Jenkins job and shared with the Beam community
through the dev list.

(2.a) Beam components should define dependencies and their versions at the
top level. There can be rare exceptions, but they should come with
explanations.

(3.a) A significantly outdated dependency (identified manually or through
the automated Jenkins job) should result in a JIRA that is a blocker for
the next release. Release manager may choose to push the blocker to the
subsequent release or downgrade from a blocker.

(4.a) Dependency declarations may identify owners that are responsible for
upgrading the respective dependencies.

(5.a) Dependencies of Java SDK components that may cause issues to other
components if leaked should be vendored.


Thanks,
Cham


>
> Kenn
>
>
>
>> Please vote:
>>> [ ] +1, Approve that we adapt these policies
>>> [ ] -1, Do not approve (please provide specific comments)
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>>> [2]
>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>
>>
>>

Re: [VOTE] Policies for managing Beam dependencies

[Kenneth Knowles <kl...@google.com>]

+0.5

I like the spirit of these policies. I think they need a little wording
work. Comments inline.

On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <ch...@google.com>
> wrote:
>>
>>
>> (1) Human readable reports on status of Beam dependencies are generated
>> weekly and shared with the Beam community through the dev list.
>>
>
Who is responsible for generating these? The mechanism or responsibility
should be made clear.

I clicked through a doc -> thread -> doc to find even some details. It
looks like manual run of a gradle command was adopted. So the
responsibility needs an owner, even if it is "unspecified volunteer on dev@
and feel free to complain or do it yourself if you don't see it"


(2) Beam components should define dependencies and their versions at the
>> top level.
>>
>
I think the big "should" works better with some guidance about when
something might be an exception, or at least explicit mention that there
can be rare exceptions. Unless you think that is never the case. If there
are no exceptions, then say "must" and if we hit a roadblock we can revisit
the policy.


(3) A significantly outdated dependency (identified manually or through
>> tooling) should result in a JIRA that is a blocker for the next release.
>> Release manager may choose to push the blocker to the subsequent release or
>> downgrade from a blocker.
>>
>
How is "significantly outdated" defined? By dev@ discussion? Seems like the
right way. Anyhow that's what will happen in practice as people debate the
blocker bug.


(4) Dependency declarations may identify owners that are responsible for
>> upgrading the respective dependencies.
>>
>> (5) Dependencies of Java SDK components that may cause issues to other
>> components if leaked should be shaded.
>>
>
We previously agreed upon our intent to migrate to "pre-shaded" aka
"vendored" packages:
https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E

With Maven, this involved a lot of boilerplate so I never did it. With
Gradle, we can easily build a re-usable rule to create such a package in a
couple of lines. I just opened the first WIP PR here:
https://github.com/apache/beam/pull/5570 it is blocked by deleting the poms
anyhow so by then we should have a configuration that works to vendor our
currently shaded artifacts.

So I think this should be rephrased to "should be vendored" so we don't
have to revise the policy.

Kenn



> Please vote:
>> [ ] +1, Approve that we adapt these policies
>> [ ] -1, Do not approve (please provide specific comments)
>>
>> Thanks,
>> Cham
>>
>> [1]
>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
>> [2]
>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>
>
>

Re: [VOTE] Policies for managing Beam dependencies

[Ahmet Altay <al...@google.com>]

+1

Thank you for driving these decisions. I would make a meta-point, all other
recent votes and if passes this one could be converted to web site
documents at some point in an easily accessible and linkable way.

On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath <ch...@google.com>
wrote:

> Hi All,
>
> We recently had a discussion regarding managing Beam dependencies. Please
> see [1] for the email thread and [2] for the relevant document.
>
> This discussion resulted in following policies. I believe, these will help
> keep Beam at a healthy state while allowing human intervention when needed.
>
> (1) Human readable reports on status of Beam dependencies are generated
> weekly and shared with the Beam community through the dev list.
>
> (2) Beam components should define dependencies and their versions at the
> top level.
>
> (3) A significantly outdated dependency (identified manually or through
> tooling) should result in a JIRA that is a blocker for the next release.
> Release manager may choose to push the blocker to the subsequent release or
> downgrade from a blocker.
>
> (4) Dependency declarations may identify owners that are responsible for
> upgrading the respective dependencies.
>
> (5) Dependencies of Java SDK components that may cause issues to other
> components if leaked should be shaded.
>
>
> Please vote:
> [ ] +1, Approve that we adapt these policies
> [ ] -1, Do not approve (please provide specific comments)
>
> Thanks,
> Cham
>
> [1] https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f8
> 09e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
> [2] https://docs.google.com/document/d/15m1MziZ5TNd9rh_
> XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>