You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Raymond Xu <xu...@gmail.com> on 2021/09/04 01:42:01 UTC
[ANNOUNCEMENT] CI changes
Hi all,
As you may have noticed, we have been running Azure Pipelines for the tests
for some time and have recently retired Travis CI in this PR
<https://github.com/apache/hudi/pull/3556>.
Background
It was a pain for the CI process in the past with Travis, which from time
to time queued up CI jobs forever. This severely affects the developer
experience for making contributions, and also the release process.
The New Setup
Thanks to the Flink community, who pioneered the CI setup, and MS Azure,
who provided the free resources, we are able to mirror the repo and PRs to
a separate GitHub organization <https://github.com/apachehudi-ci/> and run
the tests in Azure Pipelines. Hudi's ci-bot
<https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot
<https://github.com/flink-ci/ci-bot>) runs on a GCP server and periodically
scans recently changed PRs for CI submission. CI results are commented back
to the PR by hudi-bot <https://github.com/hudi-bot>. Full details about the
setup are documented in this
<https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure>
page
<https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure>
.
Azure Pipelines provides 10 free managed parallel jobs. CI tests are split
into 5 jobs. We have dedicated resources to test 2 PRs in parallel.
- master builds:
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3
- branch builds:
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5
Note: PRs against asf-site (website updates) will be ignored by this setup.
Additionally, we make use of GitHub Actions to build against different
Spark and Scala versions. GitHub Actions jobs also provide fast feedback
for compliance like checkstyle and apache-rat.
For PR Owners and Reviewers
With these changes, PR owners and reviewers should pay attention to the
following:
- CI results are indicated in hudi-bot's comment
- A new commit in the same PR will trigger a new build and cancel any
existing build
- Comment `@hudi-bot run azure` to manually trigger a new build
- GitHub Actions jobs will show as checks in the PR
- Minimum conditions to merge:
- Azure CI report shows success, and
- GitHub Actions jobs passed
- For website update PRs (for asf-site branch), owners post screenshots
to show the changes in lieu of CI tests.
Hope this contributes towards a more seamless developer experience. Please
reach out to the community for CI issues or further questions.
Best,
Raymond
Re: [ANNOUNCEMENT] CI changes
Posted by Vinoth Chandar <vi...@apache.org>.
+1 this is a truly champion effort.
On Tue, Sep 7, 2021 at 11:51 AM Sivabalan <n....@gmail.com> wrote:
> Really great job Raymond! Good to see improvements on CI infra. Definitely
> helps developer experience a lot better.
>
>
> On Mon, Sep 6, 2021 at 9:19 PM vino yang <ya...@gmail.com> wrote:
>
> > awesome! Great job!
> >
> > Thanks for driving and landing this big infra improvement!
> >
> > Best,
> > Vino
> >
> > Raymond Xu <xu...@gmail.com> 于2021年9月4日周六 上午9:42写道：
> >
> > > Hi all,
> > >
> > > As you may have noticed, we have been running Azure Pipelines for the
> > tests
> > > for some time and have recently retired Travis CI in this PR
> > > <https://github.com/apache/hudi/pull/3556>.
> > >
> > > Background
> > >
> > > It was a pain for the CI process in the past with Travis, which from
> time
> > > to time queued up CI jobs forever. This severely affects the developer
> > > experience for making contributions, and also the release process.
> > >
> > > The New Setup
> > >
> > > Thanks to the Flink community, who pioneered the CI setup, and MS
> Azure,
> > > who provided the free resources, we are able to mirror the repo and PRs
> > to
> > > a separate GitHub organization <https://github.com/apachehudi-ci/> and
> > run
> > > the tests in Azure Pipelines. Hudi's ci-bot
> > > <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot
> > > <https://github.com/flink-ci/ci-bot>) runs on a GCP server and
> > > periodically
> > > scans recently changed PRs for CI submission. CI results are commented
> > back
> > > to the PR by hudi-bot <https://github.com/hudi-bot>. Full details
> about
> > > the
> > > setup are documented in this
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> > > >
> > > page
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> > > >
> > > .
> > >
> > > Azure Pipelines provides 10 free managed parallel jobs. CI tests are
> > split
> > > into 5 jobs. We have dedicated resources to test 2 PRs in parallel.
> > >
> > > - master builds:
> > >
> > >
> >
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3
> > > - branch builds:
> > >
> > >
> >
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5
> > >
> > > Note: PRs against asf-site (website updates) will be ignored by this
> > setup.
> > >
> > > Additionally, we make use of GitHub Actions to build against different
> > > Spark and Scala versions. GitHub Actions jobs also provide fast
> feedback
> > > for compliance like checkstyle and apache-rat.
> > >
> > > For PR Owners and Reviewers
> > >
> > > With these changes, PR owners and reviewers should pay attention to the
> > > following:
> > >
> > > - CI results are indicated in hudi-bot's comment
> > > - A new commit in the same PR will trigger a new build and cancel
> any
> > > existing build
> > > - Comment `@hudi-bot run azure` to manually trigger a new build
> > > - GitHub Actions jobs will show as checks in the PR
> > > - Minimum conditions to merge:
> > > - Azure CI report shows success, and
> > > - GitHub Actions jobs passed
> > > - For website update PRs (for asf-site branch), owners post
> > screenshots
> > > to show the changes in lieu of CI tests.
> > >
> > >
> > > Hope this contributes towards a more seamless developer experience.
> > Please
> > > reach out to the community for CI issues or further questions.
> > >
> > >
> > > Best,
> > > Raymond
> > >
> >
>
>
> --
> Regards,
> -Sivabalan
>
Re: [ANNOUNCEMENT] CI changes
Posted by Sivabalan <n....@gmail.com>.
Really great job Raymond! Good to see improvements on CI infra. Definitely
helps developer experience a lot better.
On Mon, Sep 6, 2021 at 9:19 PM vino yang <ya...@gmail.com> wrote:
> awesome! Great job!
>
> Thanks for driving and landing this big infra improvement!
>
> Best,
> Vino
>
> Raymond Xu <xu...@gmail.com> 于2021年9月4日周六 上午9:42写道：
>
> > Hi all,
> >
> > As you may have noticed, we have been running Azure Pipelines for the
> tests
> > for some time and have recently retired Travis CI in this PR
> > <https://github.com/apache/hudi/pull/3556>.
> >
> > Background
> >
> > It was a pain for the CI process in the past with Travis, which from time
> > to time queued up CI jobs forever. This severely affects the developer
> > experience for making contributions, and also the release process.
> >
> > The New Setup
> >
> > Thanks to the Flink community, who pioneered the CI setup, and MS Azure,
> > who provided the free resources, we are able to mirror the repo and PRs
> to
> > a separate GitHub organization <https://github.com/apachehudi-ci/> and
> run
> > the tests in Azure Pipelines. Hudi's ci-bot
> > <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot
> > <https://github.com/flink-ci/ci-bot>) runs on a GCP server and
> > periodically
> > scans recently changed PRs for CI submission. CI results are commented
> back
> > to the PR by hudi-bot <https://github.com/hudi-bot>. Full details about
> > the
> > setup are documented in this
> > <
> >
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> > >
> > page
> > <
> >
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> > >
> > .
> >
> > Azure Pipelines provides 10 free managed parallel jobs. CI tests are
> split
> > into 5 jobs. We have dedicated resources to test 2 PRs in parallel.
> >
> > - master builds:
> >
> >
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3
> > - branch builds:
> >
> >
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5
> >
> > Note: PRs against asf-site (website updates) will be ignored by this
> setup.
> >
> > Additionally, we make use of GitHub Actions to build against different
> > Spark and Scala versions. GitHub Actions jobs also provide fast feedback
> > for compliance like checkstyle and apache-rat.
> >
> > For PR Owners and Reviewers
> >
> > With these changes, PR owners and reviewers should pay attention to the
> > following:
> >
> > - CI results are indicated in hudi-bot's comment
> > - A new commit in the same PR will trigger a new build and cancel any
> > existing build
> > - Comment `@hudi-bot run azure` to manually trigger a new build
> > - GitHub Actions jobs will show as checks in the PR
> > - Minimum conditions to merge:
> > - Azure CI report shows success, and
> > - GitHub Actions jobs passed
> > - For website update PRs (for asf-site branch), owners post
> screenshots
> > to show the changes in lieu of CI tests.
> >
> >
> > Hope this contributes towards a more seamless developer experience.
> Please
> > reach out to the community for CI issues or further questions.
> >
> >
> > Best,
> > Raymond
> >
>
--
Regards,
-Sivabalan
Re: [ANNOUNCEMENT] CI changes
Posted by vino yang <ya...@gmail.com>.
awesome! Great job!
Thanks for driving and landing this big infra improvement!
Best,
Vino
Raymond Xu <xu...@gmail.com> 于2021年9月4日周六 上午9:42写道：
> Hi all,
>
> As you may have noticed, we have been running Azure Pipelines for the tests
> for some time and have recently retired Travis CI in this PR
> <https://github.com/apache/hudi/pull/3556>.
>
> Background
>
> It was a pain for the CI process in the past with Travis, which from time
> to time queued up CI jobs forever. This severely affects the developer
> experience for making contributions, and also the release process.
>
> The New Setup
>
> Thanks to the Flink community, who pioneered the CI setup, and MS Azure,
> who provided the free resources, we are able to mirror the repo and PRs to
> a separate GitHub organization <https://github.com/apachehudi-ci/> and run
> the tests in Azure Pipelines. Hudi's ci-bot
> <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot
> <https://github.com/flink-ci/ci-bot>) runs on a GCP server and
> periodically
> scans recently changed PRs for CI submission. CI results are commented back
> to the PR by hudi-bot <https://github.com/hudi-bot>. Full details about
> the
> setup are documented in this
> <
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> >
> page
> <
> https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure
> >
> .
>
> Azure Pipelines provides 10 free managed parallel jobs. CI tests are split
> into 5 jobs. We have dedicated resources to test 2 PRs in parallel.
>
> - master builds:
>
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3
> - branch builds:
>
> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5
>
> Note: PRs against asf-site (website updates) will be ignored by this setup.
>
> Additionally, we make use of GitHub Actions to build against different
> Spark and Scala versions. GitHub Actions jobs also provide fast feedback
> for compliance like checkstyle and apache-rat.
>
> For PR Owners and Reviewers
>
> With these changes, PR owners and reviewers should pay attention to the
> following:
>
> - CI results are indicated in hudi-bot's comment
> - A new commit in the same PR will trigger a new build and cancel any
> existing build
> - Comment `@hudi-bot run azure` to manually trigger a new build
> - GitHub Actions jobs will show as checks in the PR
> - Minimum conditions to merge:
> - Azure CI report shows success, and
> - GitHub Actions jobs passed
> - For website update PRs (for asf-site branch), owners post screenshots
> to show the changes in lieu of CI tests.
>
>
> Hope this contributes towards a more seamless developer experience. Please
> reach out to the community for CI issues or further questions.
>
>
> Best,
> Raymond
>