You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Scott Wegner <sw...@google.com> on 2018/05/29 23:32:26 UTC
Design Proposal: Beam-Site Automation Reliability
I've been looking into the beam-site merge automation reliability, and I'd
like to get some early feedback on ideas for improvement. Please take a
look at https://s.apache.org/beam-site-automation:
> Apache Beam's website is maintained via the beam-site Git repository,
with a set of automation that manages the workflow from merging a pull
request to publishing. The automation is centralized in a tool called
Mergebot, which was built for Beam and donated to the ASF. However, the
automation has been somewhat unreliable, and when there are issues, very
few individuals have the necessary permissions and expertise to resolve
them. Overall, the reliability of Beam-site automation is impeding
productivity for Beam-site development.
At this point I'm seeking feedback on a few possible solutions:
1. Invest in improvements to Mergebot reliability. Make stability tweaks
for various failure modes, distribute Mergebot expertise and operations
permissions to more committers.
2. Deprecate Mergebot and revert to manual process. With the current
unreliability, some committers choose to forego merge automation anyway.
3. Generate HTML only during publishing. This seems to be newly supported
by the Apache GitPubSub workflow. This would eliminate most or all of the
automation that Mergebot is responsible for.
Feel free to add comments in the doc.
Thanks,
Scott
Got feedback? tinyurl.com/swegner-feedback
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Udi Meiri <eh...@google.com>.
FYI, we are about to add a new branch to apache/beam, named 'asf-site',
which will contain generated website sources.
On Thu, Jun 7, 2018 at 10:18 AM Jason Kuster <ja...@google.com> wrote:
> Sounds good; I'm really excited about these changes Scott. Thanks for
> taking this on!
>
> On Tue, Jun 5, 2018 at 4:00 PM Scott Wegner <sw...@google.com> wrote:
>
>> Thanks everyone; I've responded to feedback in the doc [1] and I believe
>> we've reached consensus. I've added implementation tasks in JIRA
>> under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
>> plan is:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> [1] https://s.apache.org/beam-site-automation
>> [2] https://issues.apache.org/jira/browse/BEAM-4493
>>
>> On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> Pre-commit filtering has come up on previous discussions as well and is
>>> an obvious improvement. I've opened BEAM-4445 [1] for this and assigned it
>>> to myself.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-4445
>>>
>>> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> +1
>>>>
>>>> Can we separate precommit filtering and get it set up independent from
>>>> this? I think there's a lot of good directions to go once it is the norm.
>>>>
>>>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Very nice, enthusiastic +1
>>>>>
>>>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who reviewed the doc. I put together a plan based
>>>>>> on the initial feedback to improve website automation reliability. At a
>>>>>> glance, I am proposing to:
>>>>>>
>>>>>> * Migrate website source code to the main apache/beam repository
>>>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>>>> * Align to the existing apache/beam PR process (code review policy,
>>>>>> precommits, generic Git merge)
>>>>>> * Filter pre-commit jobs to only run when necessary
>>>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>>>> publishing branch
>>>>>>
>>>>>> Please take another look at the doc, specifically the new section
>>>>>> entitled "Proposed Solution":
>>>>>> https://s.apache.org/beam-site-automation
>>>>>> I'd like to gather feedback by Monday June 4, and if there is
>>>>>> consensus move forward with the implementation.
>>>>>>
>>>>>> Thanks,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>
>>>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've been looking into the beam-site merge automation reliability,
>>>>>>> and I'd like to get some early feedback on ideas for improvement. Please
>>>>>>> take a look at https://s.apache.org/beam-site-automation:
>>>>>>>
>>>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>>>> repository, with a set of automation that manages the workflow from merging
>>>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>>>> very few individuals have the necessary permissions and expertise to
>>>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>>>> productivity for Beam-site development.
>>>>>>>
>>>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>>>
>>>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>>>> operations permissions to more committers.
>>>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>>>> all of the automation that Mergebot is responsible for.
>>>>>>>
>>>>>>> Feel free to add comments in the doc.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>>
>>>>>>
>>>>>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> <https://goto.google.com/jasonkuster-feedback>
>
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Jason Kuster <ja...@google.com>.
Sounds good; I'm really excited about these changes Scott. Thanks for
taking this on!
On Tue, Jun 5, 2018 at 4:00 PM Scott Wegner <sw...@google.com> wrote:
> Thanks everyone; I've responded to feedback in the doc [1] and I believe
> we've reached consensus. I've added implementation tasks in JIRA
> under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
> plan is:
>
> * Migrate website source code to the main apache/beam repository
> * Discontinue checking-in generated HTML during the PR workflow
> * Align to the existing apache/beam PR process (code review policy,
> precommits, generic Git merge)
> * Filter pre-commit jobs to only run when necessary
> * Add a post-commit Jenkins job to push generated HTML to a separate
> publishing branch
>
> [1] https://s.apache.org/beam-site-automation
> [2] https://issues.apache.org/jira/browse/BEAM-4493
>
> On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
>
>> Pre-commit filtering has come up on previous discussions as well and is
>> an obvious improvement. I've opened BEAM-4445 [1] for this and assigned it
>> to myself.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-4445
>>
>> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> +1
>>>
>>> Can we separate precommit filtering and get it set up independent from
>>> this? I think there's a lot of good directions to go once it is the norm.
>>>
>>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Very nice, enthusiastic +1
>>>>
>>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who reviewed the doc. I put together a plan based
>>>>> on the initial feedback to improve website automation reliability. At a
>>>>> glance, I am proposing to:
>>>>>
>>>>> * Migrate website source code to the main apache/beam repository
>>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>>> * Align to the existing apache/beam PR process (code review policy,
>>>>> precommits, generic Git merge)
>>>>> * Filter pre-commit jobs to only run when necessary
>>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>>> publishing branch
>>>>>
>>>>> Please take another look at the doc, specifically the new section
>>>>> entitled "Proposed Solution":
>>>>> https://s.apache.org/beam-site-automation
>>>>> I'd like to gather feedback by Monday June 4, and if there is
>>>>> consensus move forward with the implementation.
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>>
>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>
>>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I've been looking into the beam-site merge automation reliability,
>>>>>> and I'd like to get some early feedback on ideas for improvement. Please
>>>>>> take a look at https://s.apache.org/beam-site-automation:
>>>>>>
>>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>>> repository, with a set of automation that manages the workflow from merging
>>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>>> very few individuals have the necessary permissions and expertise to
>>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>>> productivity for Beam-site development.
>>>>>>
>>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>>
>>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>>> operations permissions to more committers.
>>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>>> all of the automation that Mergebot is responsible for.
>>>>>>
>>>>>> Feel free to add comments in the doc.
>>>>>>
>>>>>> Thanks,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>>
>>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>>
>>>>>
>>>>
--
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow
See something? Say something. go/jasonkuster-feedback
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Scott Wegner <sw...@google.com>.
Thanks everyone; I've responded to feedback in the doc [1] and I believe
we've reached consensus. I've added implementation tasks in JIRA
under BEAM-4493 [2] and will start coding soon. As a recap, the high-level
plan is:
* Migrate website source code to the main apache/beam repository
* Discontinue checking-in generated HTML during the PR workflow
* Align to the existing apache/beam PR process (code review policy,
precommits, generic Git merge)
* Filter pre-commit jobs to only run when necessary
* Add a post-commit Jenkins job to push generated HTML to a separate
publishing branch
[1] https://s.apache.org/beam-site-automation
[2] https://issues.apache.org/jira/browse/BEAM-4493
On Fri, Jun 1, 2018 at 10:33 AM Scott Wegner <sw...@google.com> wrote:
> Pre-commit filtering has come up on previous discussions as well and is an
> obvious improvement. I've opened BEAM-4445 [1] for this and assigned it to
> myself.
>
> [1] https://issues.apache.org/jira/browse/BEAM-4445
>
> On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> +1
>>
>> Can we separate precommit filtering and get it set up independent from
>> this? I think there's a lot of good directions to go once it is the norm.
>>
>> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> Very nice, enthusiastic +1
>>>
>>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com>
>>> wrote:
>>>
>>>> Thanks to everyone who reviewed the doc. I put together a plan based on
>>>> the initial feedback to improve website automation reliability. At a
>>>> glance, I am proposing to:
>>>>
>>>> * Migrate website source code to the main apache/beam repository
>>>> * Discontinue checking-in generated HTML during the PR workflow
>>>> * Align to the existing apache/beam PR process (code review policy,
>>>> precommits, generic Git merge)
>>>> * Filter pre-commit jobs to only run when necessary
>>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>>> publishing branch
>>>>
>>>> Please take another look at the doc, specifically the new section
>>>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>>>>
>>>> I'd like to gather feedback by Monday June 4, and if there is consensus
>>>> move forward with the implementation.
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>>
>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>
>>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> I've been looking into the beam-site merge automation reliability, and
>>>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>>>> look at https://s.apache.org/beam-site-automation:
>>>>>
>>>>> > Apache Beam's website is maintained via the beam-site Git
>>>>> repository, with a set of automation that manages the workflow from merging
>>>>> a pull request to publishing. The automation is centralized in a tool
>>>>> called Mergebot, which was built for Beam and donated to the ASF. However,
>>>>> the automation has been somewhat unreliable, and when there are issues,
>>>>> very few individuals have the necessary permissions and expertise to
>>>>> resolve them. Overall, the reliability of Beam-site automation is impeding
>>>>> productivity for Beam-site development.
>>>>>
>>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>>
>>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>>> operations permissions to more committers.
>>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>>> unreliability, some committers choose to forego merge automation anyway.
>>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>>> all of the automation that Mergebot is responsible for.
>>>>>
>>>>> Feel free to add comments in the doc.
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>>
>>>>>
>>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>>
>>>>
>>>
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Scott Wegner <sw...@google.com>.
Pre-commit filtering has come up on previous discussions as well and is an
obvious improvement. I've opened BEAM-4445 [1] for this and assigned it to
myself.
[1] https://issues.apache.org/jira/browse/BEAM-4445
On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles <kl...@google.com> wrote:
> +1
>
> Can we separate precommit filtering and get it set up independent from
> this? I think there's a lot of good directions to go once it is the norm.
>
> On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
>
>> Very nice, enthusiastic +1
>>
>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:
>>
>>> Thanks to everyone who reviewed the doc. I put together a plan based on
>>> the initial feedback to improve website automation reliability. At a
>>> glance, I am proposing to:
>>>
>>> * Migrate website source code to the main apache/beam repository
>>> * Discontinue checking-in generated HTML during the PR workflow
>>> * Align to the existing apache/beam PR process (code review policy,
>>> precommits, generic Git merge)
>>> * Filter pre-commit jobs to only run when necessary
>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>> publishing branch
>>>
>>> Please take another look at the doc, specifically the new section
>>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>>> I'd like to gather feedback by Monday June 4, and if there is consensus
>>> move forward with the implementation.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>>>
>>>> I've been looking into the beam-site merge automation reliability, and
>>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>>> look at https://s.apache.org/beam-site-automation:
>>>>
>>>> > Apache Beam's website is maintained via the beam-site Git repository,
>>>> with a set of automation that manages the workflow from merging a pull
>>>> request to publishing. The automation is centralized in a tool called
>>>> Mergebot, which was built for Beam and donated to the ASF. However, the
>>>> automation has been somewhat unreliable, and when there are issues, very
>>>> few individuals have the necessary permissions and expertise to resolve
>>>> them. Overall, the reliability of Beam-site automation is impeding
>>>> productivity for Beam-site development.
>>>>
>>>> At this point I'm seeking feedback on a few possible solutions:
>>>>
>>>> 1. Invest in improvements to Mergebot reliability. Make stability
>>>> tweaks for various failure modes, distribute Mergebot expertise and
>>>> operations permissions to more committers.
>>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>>> unreliability, some committers choose to forego merge automation anyway.
>>>> 3. Generate HTML only during publishing. This seems to be newly
>>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>>> all of the automation that Mergebot is responsible for.
>>>>
>>>> Feel free to add comments in the doc.
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>>
>>>>
>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>
>>>
>>
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Kenneth Knowles <kl...@google.com>.
+1
Can we separate precommit filtering and get it set up independent from
this? I think there's a lot of good directions to go once it is the norm.
On Thu, May 31, 2018 at 9:25 PM Thomas Weise <th...@apache.org> wrote:
> Very nice, enthusiastic +1
>
> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:
>
>> Thanks to everyone who reviewed the doc. I put together a plan based on
>> the initial feedback to improve website automation reliability. At a
>> glance, I am proposing to:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> Please take another look at the doc, specifically the new section
>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>> I'd like to gather feedback by Monday June 4, and if there is consensus
>> move forward with the implementation.
>>
>> Thanks,
>> Scott
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>>
>>> I've been looking into the beam-site merge automation reliability, and
>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>> look at https://s.apache.org/beam-site-automation:
>>>
>>> > Apache Beam's website is maintained via the beam-site Git repository,
>>> with a set of automation that manages the workflow from merging a pull
>>> request to publishing. The automation is centralized in a tool called
>>> Mergebot, which was built for Beam and donated to the ASF. However, the
>>> automation has been somewhat unreliable, and when there are issues, very
>>> few individuals have the necessary permissions and expertise to resolve
>>> them. Overall, the reliability of Beam-site automation is impeding
>>> productivity for Beam-site development.
>>>
>>> At this point I'm seeking feedback on a few possible solutions:
>>>
>>> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
>>> for various failure modes, distribute Mergebot expertise and operations
>>> permissions to more committers.
>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>> unreliability, some committers choose to forego merge automation anyway.
>>> 3. Generate HTML only during publishing. This seems to be newly
>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>> all of the automation that Mergebot is responsible for.
>>>
>>> Feel free to add comments in the doc.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>
>
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Thomas Weise <th...@apache.org>.
Very nice, enthusiastic +1
On Thu, May 31, 2018 at 3:24 PM, Scott Wegner <sw...@google.com> wrote:
> Thanks to everyone who reviewed the doc. I put together a plan based on
> the initial feedback to improve website automation reliability. At a
> glance, I am proposing to:
>
> * Migrate website source code to the main apache/beam repository
> * Discontinue checking-in generated HTML during the PR workflow
> * Align to the existing apache/beam PR process (code review policy,
> precommits, generic Git merge)
> * Filter pre-commit jobs to only run when necessary
> * Add a post-commit Jenkins job to push generated HTML to a separate
> publishing branch
>
> Please take another look at the doc, specifically the new section entitled
> "Proposed Solution": https://s.apache.org/beam-site-automation
> I'd like to gather feedback by Monday June 4, and if there is consensus
> move forward with the implementation.
>
> Thanks,
> Scott
>
>
> Got feedback? tinyurl.com/swegner-feedback
>
> On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
>
>> I've been looking into the beam-site merge automation reliability, and
>> I'd like to get some early feedback on ideas for improvement. Please take a
>> look at https://s.apache.org/beam-site-automation:
>>
>> > Apache Beam's website is maintained via the beam-site Git repository,
>> with a set of automation that manages the workflow from merging a pull
>> request to publishing. The automation is centralized in a tool called
>> Mergebot, which was built for Beam and donated to the ASF. However, the
>> automation has been somewhat unreliable, and when there are issues, very
>> few individuals have the necessary permissions and expertise to resolve
>> them. Overall, the reliability of Beam-site automation is impeding
>> productivity for Beam-site development.
>>
>> At this point I'm seeking feedback on a few possible solutions:
>>
>> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
>> for various failure modes, distribute Mergebot expertise and operations
>> permissions to more committers.
>> 2. Deprecate Mergebot and revert to manual process. With the current
>> unreliability, some committers choose to forego merge automation anyway.
>> 3. Generate HTML only during publishing. This seems to be newly supported
>> by the Apache GitPubSub workflow. This would eliminate most or all of the
>> automation that Mergebot is responsible for.
>>
>> Feel free to add comments in the doc.
>>
>> Thanks,
>> Scott
>>
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>
Re: Design Proposal: Beam-Site Automation Reliability
Posted by Scott Wegner <sw...@google.com>.
Thanks to everyone who reviewed the doc. I put together a plan based on the
initial feedback to improve website automation reliability. At a glance, I
am proposing to:
* Migrate website source code to the main apache/beam repository
* Discontinue checking-in generated HTML during the PR workflow
* Align to the existing apache/beam PR process (code review policy,
precommits, generic Git merge)
* Filter pre-commit jobs to only run when necessary
* Add a post-commit Jenkins job to push generated HTML to a separate
publishing branch
Please take another look at the doc, specifically the new section entitled
"Proposed Solution": https://s.apache.org/beam-site-automation
I'd like to gather feedback by Monday June 4, and if there is consensus
move forward with the implementation.
Thanks,
Scott
Got feedback? tinyurl.com/swegner-feedback
On Tue, May 29, 2018 at 4:32 PM Scott Wegner <sw...@google.com> wrote:
> I've been looking into the beam-site merge automation reliability, and I'd
> like to get some early feedback on ideas for improvement. Please take a
> look at https://s.apache.org/beam-site-automation:
>
> > Apache Beam's website is maintained via the beam-site Git repository,
> with a set of automation that manages the workflow from merging a pull
> request to publishing. The automation is centralized in a tool called
> Mergebot, which was built for Beam and donated to the ASF. However, the
> automation has been somewhat unreliable, and when there are issues, very
> few individuals have the necessary permissions and expertise to resolve
> them. Overall, the reliability of Beam-site automation is impeding
> productivity for Beam-site development.
>
> At this point I'm seeking feedback on a few possible solutions:
>
> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
> for various failure modes, distribute Mergebot expertise and operations
> permissions to more committers.
> 2. Deprecate Mergebot and revert to manual process. With the current
> unreliability, some committers choose to forego merge automation anyway.
> 3. Generate HTML only during publishing. This seems to be newly supported
> by the Apache GitPubSub workflow. This would eliminate most or all of the
> automation that Mergebot is responsible for.
>
> Feel free to add comments in the doc.
>
> Thanks,
> Scott
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>