Re: Design Proposal: Beam-Site Automation Reliability

[Thomas Weise <th...@apache.org>]

Very nice, enthusiastic +1

On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:

> Thanks to everyone who reviewed the doc. I put together a plan based on
> the initial feedback to improve website automation reliability. At a
> glance, I am proposing to:
>
> * Migrate website source code to the main apache/beam repository
> * Discontinue checking-in generated HTML during the PR workflow
> * Align to the existing apache/beam PR process (code review policy,
> precommits, generic Git merge)
> * Filter pre-commit jobs to only run when necessary
> * Add a post-commit Jenkins job to push generated HTML to a separate
> publishing branch
>
> Please take another look at the doc, specifically the new section entitled
> "Proposed Solution": https://s.apache.org/beam-site-automation
> I'd like to gather feedback by Monday June 4, and if there is consensus
> move forward with the implementation.
>
> Thanks,
> Scott
>
>
> Got feedback? tinyurl.com/swegner-feedback
>
> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>
>> I've been looking into the beam-site merge automation reliability, and
>> I'd like to get some early feedback on ideas for improvement. Please take a
>> look at https://s.apache.org/beam-site-automation:
>>
>> > Apache Beam's website is maintained via the beam-site Git repository,
>> with a set of automation that manages the workflow from merging a pull
>> request to publishing. The automation is centralized in a tool called
>> Mergebot, which was built for Beam and donated to the ASF. However, the
>> automation has been somewhat unreliable, and when there are issues, very
>> few individuals have the necessary permissions and expertise to resolve
>> them. Overall, the reliability of Beam-site automation is impeding
>> productivity for Beam-site development.
>>
>> At this point I'm seeking feedback on a few possible solutions:
>>
>> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
>> for various failure modes, distribute Mergebot expertise and operations
>> permissions to more committers.
>> 2. Deprecate Mergebot and revert to manual process. With the current
>> unreliability, some committers choose to forego merge automation anyway.
>> 3. Generate HTML only during publishing. This seems to be newly supported
>> by the Apache GitPubSub workflow. This would eliminate most or all of the
>> automation that Mergebot is responsible for.
>>
>> Feel free to add comments in the doc.
>>
>> Thanks,
>> Scott
>>
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>

Re: Design Proposal: Beam-Site Automation Reliability

[Udi Meiri <eh...@google.com>]

FYI, we are about to add a new branch to apache/beam, named 'asf-site',
which will contain generated website sources.

On Thu, Jun 7, 2018 at 10:18 AM Jason Kuster <ja...@google.com> wrote:

> Sounds good; I'm really excited about these changes Scott. Thanks for
> taking this on!
>
> On Tue, Jun 5, 2018 at 4:00 PM Scott Wegner <sw...@google.com> wrote:
>
>> Thanks everyone; I've responded to feedback in the doc [1] and I believe
>> we've reached consensus. I've added implementation tasks in JIRA
>> under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
>> plan is:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> [1] https://s.apache.org/beam-site-automation
>> [2] https://issues.apache.org/jira/browse/BEAM-4493
>>
>> On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> Pre-commit filtering has come up on previous discussions as well and is
>>> an obvious improvement. I've opened BEAM-4445 [1] for this and assigned it
>>> to myself.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-4445
>>>
>>> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> +1
>>>>
>>>> Can we separate precommit filtering and get it set up independent from
>>>> this? I think there's a lot of good directions to go once it is the norm.
>>>>
>>>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Very nice, enthusiastic +1
>>>>>
>>>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who reviewed the doc. I put together a plan based
>>>>>> on the initial feedback to improve website automation reliability. At a
>>>>>> glance, I am proposing to:
>>>>>>
>>>>>> * Migrate website source code to the main apache/beam repository
>>>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>>>> * Align to the existing apache/beam PR process (code review policy,
>>>>>> precommits, generic Git merge)
>>>>>> * Filter pre-commit jobs to only run when necessary
>>>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>>>> publishing branch
>>>>>>
>>>>>> Please take another look at the doc, specifically the new section
>>>>>> entitled "Proposed Solution":
>>>>>> https://s.apache.org/beam-site-automation
>>>>>> I'd like to gather feedback by Monday June 4, and if there is
>>>>>> consensus move forward with the implementation.
>>>>>>
>>>>>> Thanks,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>
>>>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've been looking into the beam-site merge automation reliability,
>>>>>>> and I'd like to get some early feedback on ideas for improvement. Please
>>>>>>> take a look at https://s.apache.org/beam-site-automation:
>>>>>>>
>>>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>>>> repository, with a set of automation that manages the workflow from merging
>>>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>>>> very few individuals have the necessary permissions and expertise to
>>>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>>>> productivity for Beam-site development.
>>>>>>>
>>>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>>>
>>>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>>>> operations permissions to more committers.
>>>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>>>> all of the automation that Mergebot is responsible for.
>>>>>>>
>>>>>>> Feel free to add comments in the doc.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>>
>>>>>>
>>>>>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> <https://goto.google.com/jasonkuster-feedback>
>

Re: Design Proposal: Beam-Site Automation Reliability

[Jason Kuster <ja...@google.com>]

Sounds good; I'm really excited about these changes Scott. Thanks for
taking this on!

On Tue, Jun 5, 2018 at 4:00 PM Scott Wegner <sw...@google.com> wrote:

> Thanks everyone; I've responded to feedback in the doc [1] and I believe
> we've reached consensus. I've added implementation tasks in JIRA
> under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
> plan is:
>
> * Migrate website source code to the main apache/beam repository
> * Discontinue checking-in generated HTML during the PR workflow
> * Align to the existing apache/beam PR process (code review policy,
> precommits, generic Git merge)
> * Filter pre-commit jobs to only run when necessary
> * Add a post-commit Jenkins job to push generated HTML to a separate
> publishing branch
>
> [1] https://s.apache.org/beam-site-automation
> [2] https://issues.apache.org/jira/browse/BEAM-4493
>
> On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
>
>> Pre-commit filtering has come up on previous discussions as well and is
>> an obvious improvement. I've opened BEAM-4445 [1] for this and assigned it
>> to myself.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-4445
>>
>> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> +1
>>>
>>> Can we separate precommit filtering and get it set up independent from
>>> this? I think there's a lot of good directions to go once it is the norm.
>>>
>>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Very nice, enthusiastic +1
>>>>
>>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who reviewed the doc. I put together a plan based
>>>>> on the initial feedback to improve website automation reliability. At a
>>>>> glance, I am proposing to:
>>>>>
>>>>> * Migrate website source code to the main apache/beam repository
>>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>>> * Align to the existing apache/beam PR process (code review policy,
>>>>> precommits, generic Git merge)
>>>>> * Filter pre-commit jobs to only run when necessary
>>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>>> publishing branch
>>>>>
>>>>> Please take another look at the doc, specifically the new section
>>>>> entitled "Proposed Solution":
>>>>> https://s.apache.org/beam-site-automation
>>>>> I'd like to gather feedback by Monday June 4, and if there is
>>>>> consensus move forward with the implementation.
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>>
>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>
>>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I've been looking into the beam-site merge automation reliability,
>>>>>> and I'd like to get some early feedback on ideas for improvement. Please
>>>>>> take a look at https://s.apache.org/beam-site-automation:
>>>>>>
>>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>>> repository, with a set of automation that manages the workflow from merging
>>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>>> very few individuals have the necessary permissions and expertise to
>>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>>> productivity for Beam-site development.
>>>>>>
>>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>>
>>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>>> operations permissions to more committers.
>>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>>> all of the automation that Mergebot is responsible for.
>>>>>>
>>>>>> Feel free to add comments in the doc.
>>>>>>
>>>>>> Thanks,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>>
>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>
>>>>>
>>>>

-- 
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Re: Design Proposal: Beam-Site Automation Reliability

[Scott Wegner <sw...@google.com>]

Thanks everyone; I've responded to feedback in the doc [1] and I believe
we've reached consensus. I've added implementation tasks in JIRA
under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
plan is:

* Migrate website source code to the main apache/beam repository
* Discontinue checking-in generated HTML during the PR workflow
* Align to the existing apache/beam PR process (code review policy,
precommits, generic Git merge)
* Filter pre-commit jobs to only run when necessary
* Add a post-commit Jenkins job to push generated HTML to a separate
publishing branch

[1] https://s.apache.org/beam-site-automation
[2] https://issues.apache.org/jira/browse/BEAM-4493

On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:

> Pre-commit filtering has come up on previous discussions as well and is an
> obvious improvement. I've opened BEAM-4445 [1] for this and assigned it to
> myself.
>
> [1] https://issues.apache.org/jira/browse/BEAM-4445
>
> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> +1
>>
>> Can we separate precommit filtering and get it set up independent from
>> this? I think there's a lot of good directions to go once it is the norm.
>>
>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> Very nice, enthusiastic +1
>>>
>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>> wrote:
>>>
>>>> Thanks to everyone who reviewed the doc. I put together a plan based on
>>>> the initial feedback to improve website automation reliability. At a
>>>> glance, I am proposing to:
>>>>
>>>> * Migrate website source code to the main apache/beam repository
>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>> * Align to the existing apache/beam PR process (code review policy,
>>>> precommits, generic Git merge)
>>>> * Filter pre-commit jobs to only run when necessary
>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>> publishing branch
>>>>
>>>> Please take another look at the doc, specifically the new section
>>>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>>>>
>>>> I'd like to gather feedback by Monday June 4, and if there is consensus
>>>> move forward with the implementation.
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>>
>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>
>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> I've been looking into the beam-site merge automation reliability, and
>>>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>>>> look at https://s.apache.org/beam-site-automation:
>>>>>
>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>> repository, with a set of automation that manages the workflow from merging
>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>> very few individuals have the necessary permissions and expertise to
>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>> productivity for Beam-site development.
>>>>>
>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>
>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>> operations permissions to more committers.
>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>> all of the automation that Mergebot is responsible for.
>>>>>
>>>>> Feel free to add comments in the doc.
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>>
>>>>>
>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>
>>>>
>>>

Re: Design Proposal: Beam-Site Automation Reliability

[Scott Wegner <sw...@google.com>]

Pre-commit filtering has come up on previous discussions as well and is an
obvious improvement. I've opened BEAM-4445 [1] for this and assigned it to
myself.

[1] https://issues.apache.org/jira/browse/BEAM-4445

On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:

> +1
>
> Can we separate precommit filtering and get it set up independent from
> this? I think there's a lot of good directions to go once it is the norm.
>
> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>
>> Very nice, enthusiastic +1
>>
>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:
>>
>>> Thanks to everyone who reviewed the doc. I put together a plan based on
>>> the initial feedback to improve website automation reliability. At a
>>> glance, I am proposing to:
>>>
>>> * Migrate website source code to the main apache/beam repository
>>> * Discontinue checking-in generated HTML during the PR workflow
>>> * Align to the existing apache/beam PR process (code review policy,
>>> precommits, generic Git merge)
>>> * Filter pre-commit jobs to only run when necessary
>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>> publishing branch
>>>
>>> Please take another look at the doc, specifically the new section
>>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>>> I'd like to gather feedback by Monday June 4, and if there is consensus
>>> move forward with the implementation.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>>>
>>>> I've been looking into the beam-site merge automation reliability, and
>>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>>> look at https://s.apache.org/beam-site-automation:
>>>>
>>>> > Apache Beam's website is maintained via the beam-site Git repository,
>>>> with a set of automation that manages the workflow from merging a pull
>>>> request to publishing. The automation is centralized in a tool called
>>>> Mergebot, which was built for Beam and donated to the ASF. However, the
>>>> automation has been somewhat unreliable, and when there are issues, very
>>>> few individuals have the necessary permissions and expertise to resolve
>>>> them. Overall, the reliability of Beam-site automation is impeding
>>>> productivity for Beam-site development.
>>>>
>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>
>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>> operations permissions to more committers.
>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>> unreliability, some committers choose to forego merge automation anyway.
>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>> all of the automation that Mergebot is responsible for.
>>>>
>>>> Feel free to add comments in the doc.
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>>
>>>>
>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>
>>>
>>

Re: Design Proposal: Beam-Site Automation Reliability

[Kenneth Knowles <kl...@google.com>]

+1

Can we separate precommit filtering and get it set up independent from
this? I think there's a lot of good directions to go once it is the norm.

On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:

> Very nice, enthusiastic +1
>
> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:
>
>> Thanks to everyone who reviewed the doc. I put together a plan based on
>> the initial feedback to improve website automation reliability. At a
>> glance, I am proposing to:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> Please take another look at the doc, specifically the new section
>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>> I'd like to gather feedback by Monday June 4, and if there is consensus
>> move forward with the implementation.
>>
>> Thanks,
>> Scott
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>>
>>> I've been looking into the beam-site merge automation reliability, and
>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>> look at https://s.apache.org/beam-site-automation:
>>>
>>> > Apache Beam's website is maintained via the beam-site Git repository,
>>> with a set of automation that manages the workflow from merging a pull
>>> request to publishing. The automation is centralized in a tool called
>>> Mergebot, which was built for Beam and donated to the ASF. However, the
>>> automation has been somewhat unreliable, and when there are issues, very
>>> few individuals have the necessary permissions and expertise to resolve
>>> them. Overall, the reliability of Beam-site automation is impeding
>>> productivity for Beam-site development.
>>>
>>> At this point I'm seeking feedback on a few possible solutions:
>>>
>>> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
>>> for various failure modes, distribute Mergebot expertise and operations
>>> permissions to more committers.
>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>> unreliability, some committers choose to forego merge automation anyway.
>>> 3. Generate HTML only during publishing. This seems to be newly
>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>> all of the automation that Mergebot is responsible for.
>>>
>>> Feel free to add comments in the doc.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>
>