Re: Design Proposal: Beam-Site Automation Reliability

[Udi Meiri <eh...@google.com>]

FYI, we are about to add a new branch to apache/beam, named 'asf-site',
which will contain generated website sources.

On Thu, Jun 7, 2018 at 10:18 AM Jason Kuster <ja...@google.com> wrote:

> Sounds good; I'm really excited about these changes Scott. Thanks for
> taking this on!
>
> On Tue, Jun 5, 2018 at 4:00 PM Scott Wegner <sw...@google.com> wrote:
>
>> Thanks everyone; I've responded to feedback in the doc [1] and I believe
>> we've reached consensus. I've added implementation tasks in JIRA
>> under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
>> plan is:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> [1] https://s.apache.org/beam-site-automation
>> [2] https://issues.apache.org/jira/browse/BEAM-4493
>>
>> On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> Pre-commit filtering has come up on previous discussions as well and is
>>> an obvious improvement. I've opened BEAM-4445 [1] for this and assigned it
>>> to myself.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-4445
>>>
>>> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> +1
>>>>
>>>> Can we separate precommit filtering and get it set up independent from
>>>> this? I think there's a lot of good directions to go once it is the norm.
>>>>
>>>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Very nice, enthusiastic +1
>>>>>
>>>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who reviewed the doc. I put together a plan based
>>>>>> on the initial feedback to improve website automation reliability. At a
>>>>>> glance, I am proposing to:
>>>>>>
>>>>>> * Migrate website source code to the main apache/beam repository
>>>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>>>> * Align to the existing apache/beam PR process (code review policy,
>>>>>> precommits, generic Git merge)
>>>>>> * Filter pre-commit jobs to only run when necessary
>>>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>>>> publishing branch
>>>>>>
>>>>>> Please take another look at the doc, specifically the new section
>>>>>> entitled "Proposed Solution":
>>>>>> https://s.apache.org/beam-site-automation
>>>>>> I'd like to gather feedback by Monday June 4, and if there is
>>>>>> consensus move forward with the implementation.
>>>>>>
>>>>>> Thanks,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>
>>>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've been looking into the beam-site merge automation reliability,
>>>>>>> and I'd like to get some early feedback on ideas for improvement. Please
>>>>>>> take a look at https://s.apache.org/beam-site-automation:
>>>>>>>
>>>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>>>> repository, with a set of automation that manages the workflow from merging
>>>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>>>> very few individuals have the necessary permissions and expertise to
>>>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>>>> productivity for Beam-site development.
>>>>>>>
>>>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>>>
>>>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>>>> operations permissions to more committers.
>>>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>>>> all of the automation that Mergebot is responsible for.
>>>>>>>
>>>>>>> Feel free to add comments in the doc.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>>
>>>>>>
>>>>>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> <https://goto.google.com/jasonkuster-feedback>
>