You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Nick Dimiduk <nd...@apache.org> on 2026/02/06 15:29:37 UTC
Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI
Heya team,
We now have static analysis, build, and unit test coverage via GHA implemented for PRs against all branches. In the absence of comments, you'll find the detailed Yetus table reports in the "Action Summary" section of the checks interface. Implementation details are attached to HBASE-29838. As before, shout if you find something broken.
As of this effort, all of our PR work is being done in GHA. All of the Jenkins infrastructure is still present and intact, but I have disabled the job in the Jenkins UI [0]. If you're having problems with the new tooling and want to restore the Jenkins job, just flip that switch back to enabled.
There's more jobs to be converted, I've not filed any additional Jiras yet. Please feel encouraged to tackle something if you're feeling so inclined.
Thanks,
Nick
[0]: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR
On 2026/01/15 14:35:50 Nick Dimiduk wrote:
> Hi team,
>
> I've added GitHub Actions to run Yetus general checks directly on pull requests. This has been backported to all active branches (master, branch-3, branch-2, branch-2.6, branch-2.5).
>
> What's changing:
> - PRs will now trigger a "Yetus General Check" workflow that runs checkstyle, javac, pylint, shellcheck, and other static analysis checks
> - Results appear as GitHub status checks and PR comments (same as Jenkins)
> - Old bot comments from previous runs are automatically hidden to reduce noise
> - Uses the same scripts, Dockerfile, &c. that we use for Jenkins
>
> What's NOT changing:
> - Jenkins precommit builds continue to run as before
> - Your PR workflow remains the same
> - Unit/integration tests still run on Jenkins, not GitHub Actions
>
> JIRAs: HBASE-29787, HBASE-29829
>
> Give us a shout if anything breaks for you. Also, feel free to pick up migrating other jobs, if you want to (I've not filed any subsequent tickets). It's a little bit tedious and it seems that you need push rights to the origin github repo to introduce a new workflow (I'm not sure about editing an existing workflow).
>
> Thanks,
> Nick
>
> On 2024/10/02 09:22:51 Nick Dimiduk wrote:
> > Heya,
> >
> > I'd like to take the community temperature on migrating our build infra
> > from the ci-hbase.a.o Jenkins instance to something built on GitHub
> > Actions. I have several reasons that justify this proposal.
> >
> > As some of you may know, our community funding has reduced and we will no
> > longer be able to sustain the current fleet of build infrastructure. So,
> > one motivation for this proposal is cost-cutting: I think that we'll be
> > able to operate at lower costs if we can migrate to a provisioned-as-needed
> > model of consumption.
> >
> > My second reason is an optimistic appeal to a larger contributor base. I
> > suspect that if we can modernize our infrastructure then we will increase
> > the pool of contributors who might be able to participate in this area. I
> > believe that GH Actions (and systems like it) is more prevalent in the
> > industry than Jenkins, which means that more people already have experience
> > with the platform and more people will feel compelled to offer support to
> > an OSS project that uses the platform as a means of growing their own
> > skillset and as a means of bolstering their CVs.
> >
> > Dove-tailed into reason two is reason three: I believe that there is a
> > large community of folks who are developing GitHub Actions on its
> > marketplace. We would effectively open ourselves up to more off-the-shelf
> > offerings and those offerings would be in our hands directly. By contrast,
> > I don't think there's as much development in Jenkins plugins, and the
> > process of adding a new plugin to our Jenkins instance requires filing an
> > INFRA ticket.
> >
> > These are my motivations. I'm still not clear on what's possible yet for
> > ASF projects. I have filed an INFRA ticket, requesting whatever is
> > necessary for us to start an experiment. Indeed, I believe that there are
> > some major limitations on the current implementation provided by the ASF,
> > and as far as I can tell, only one project with a build footprint that
> > resembles HBase has pursued this effort. I've catalogued the applicable
> > information that I've found so far on that issue.
> >
> > https://issues.apache.org/jira/browse/INFRA-26170
> >
> > Thanks,
> > Nick
> >
>
Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
I've opened HBASER-29872 for tracking the disabling of our previous
pre commit github jenkins job.
And there is a difference that, now if a PR is not opened by a
committer, the pre commit checks workflow needs to be approved by a
committer before running, while in the past the jenkins job would scan
the github PRs and schedule pre commit jobs automatically.
Nick Dimiduk <nd...@apache.org> 于2026年2月6日周五 23:45写道：
>
> Hi Charles,
>
> I did, yes. My strategy is to break the large test set down into several
> waves. Actually, I have 5 total waves for unit tests: small, medium,
> large-1, large-2, large-3. They run in parallel and so the total time for
> feedback is much faster than the Jenkins model... at the cost of pretty
> high resource usage on branch-2*. It's possible that we'll run into issues
> with Github's generosity with the free tier runners, but we'll have to
> cross that bridge with INFRA if/when it comes up.
>
> -n
>
> On Fri, Feb 6, 2026 at 4:39 PM Charles Connell via dev <de...@hbase.apache.org>
> wrote:
>
> > Thanks for your work Nick,
> >
> > When exploring Github actions myself, I found that they might struggle
> > to run the unit tests in the hbase-server module. Any single action
> > step is limited to 6 hours runtime, and running these tests can take
> > longer than that on the action runner, in my experience. Did you work
> > around this somehow?
> >
> > Charles
> >
> > On Fri, Feb 6, 2026 at 10:29 AM Nick Dimiduk <nd...@apache.org> wrote:
> > >
> > > Heya team,
> > >
> > > We now have static analysis, build, and unit test coverage via GHA
> > implemented for PRs against all branches. In the absence of comments,
> > you'll find the detailed Yetus table reports in the "Action Summary"
> > section of the checks interface. Implementation details are attached to
> > HBASE-29838. As before, shout if you find something broken.
> > >
> > > As of this effort, all of our PR work is being done in GHA. All of the
> > Jenkins infrastructure is still present and intact, but I have disabled the
> > job in the Jenkins UI [0]. If you're having problems with the new tooling
> > and want to restore the Jenkins job, just flip that switch back to enabled.
> > >
> > > There's more jobs to be converted, I've not filed any additional Jiras
> > yet. Please feel encouraged to tackle something if you're feeling so
> > inclined.
> > >
> > > Thanks,
> > > Nick
> > >
> > > [0]: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR
> > >
> > > On 2026/01/15 14:35:50 Nick Dimiduk wrote:
> > > > Hi team,
> > > >
> > > > I've added GitHub Actions to run Yetus general checks directly on pull
> > requests. This has been backported to all active branches (master,
> > branch-3, branch-2, branch-2.6, branch-2.5).
> > > >
> > > > What's changing:
> > > > - PRs will now trigger a "Yetus General Check" workflow that runs
> > checkstyle, javac, pylint, shellcheck, and other static analysis checks
> > > > - Results appear as GitHub status checks and PR comments (same as
> > Jenkins)
> > > > - Old bot comments from previous runs are automatically hidden to
> > reduce noise
> > > > - Uses the same scripts, Dockerfile, &c. that we use for Jenkins
> > > >
> > > > What's NOT changing:
> > > > - Jenkins precommit builds continue to run as before
> > > > - Your PR workflow remains the same
> > > > - Unit/integration tests still run on Jenkins, not GitHub Actions
> > > >
> > > > JIRAs: HBASE-29787, HBASE-29829
> > > >
> > > > Give us a shout if anything breaks for you. Also, feel free to pick up
> > migrating other jobs, if you want to (I've not filed any subsequent
> > tickets). It's a little bit tedious and it seems that you need push rights
> > to the origin github repo to introduce a new workflow (I'm not sure about
> > editing an existing workflow).
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > On 2024/10/02 09:22:51 Nick Dimiduk wrote:
> > > > > Heya,
> > > > >
> > > > > I'd like to take the community temperature on migrating our build
> > infra
> > > > > from the ci-hbase.a.o Jenkins instance to something built on GitHub
> > > > > Actions. I have several reasons that justify this proposal.
> > > > >
> > > > > As some of you may know, our community funding has reduced and we
> > will no
> > > > > longer be able to sustain the current fleet of build infrastructure.
> > So,
> > > > > one motivation for this proposal is cost-cutting: I think that we'll
> > be
> > > > > able to operate at lower costs if we can migrate to a
> > provisioned-as-needed
> > > > > model of consumption.
> > > > >
> > > > > My second reason is an optimistic appeal to a larger contributor
> > base. I
> > > > > suspect that if we can modernize our infrastructure then we will
> > increase
> > > > > the pool of contributors who might be able to participate in this
> > area. I
> > > > > believe that GH Actions (and systems like it) is more prevalent in
> > the
> > > > > industry than Jenkins, which means that more people already have
> > experience
> > > > > with the platform and more people will feel compelled to offer
> > support to
> > > > > an OSS project that uses the platform as a means of growing their own
> > > > > skillset and as a means of bolstering their CVs.
> > > > >
> > > > > Dove-tailed into reason two is reason three: I believe that there is
> > a
> > > > > large community of folks who are developing GitHub Actions on its
> > > > > marketplace. We would effectively open ourselves up to more
> > off-the-shelf
> > > > > offerings and those offerings would be in our hands directly. By
> > contrast,
> > > > > I don't think there's as much development in Jenkins plugins, and the
> > > > > process of adding a new plugin to our Jenkins instance requires
> > filing an
> > > > > INFRA ticket.
> > > > >
> > > > > These are my motivations. I'm still not clear on what's possible yet
> > for
> > > > > ASF projects. I have filed an INFRA ticket, requesting whatever is
> > > > > necessary for us to start an experiment. Indeed, I believe that
> > there are
> > > > > some major limitations on the current implementation provided by the
> > ASF,
> > > > > and as far as I can tell, only one project with a build footprint
> > that
> > > > > resembles HBase has pursued this effort. I've catalogued the
> > applicable
> > > > > information that I've found so far on that issue.
> > > > >
> > > > > https://issues.apache.org/jira/browse/INFRA-26170
> > > > >
> > > > > Thanks,
> > > > > Nick
> > > > >
> > > >
> >
Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI
Posted by Nick Dimiduk <nd...@apache.org>.
Hi Charles,
I did, yes. My strategy is to break the large test set down into several
waves. Actually, I have 5 total waves for unit tests: small, medium,
large-1, large-2, large-3. They run in parallel and so the total time for
feedback is much faster than the Jenkins model... at the cost of pretty
high resource usage on branch-2*. It's possible that we'll run into issues
with Github's generosity with the free tier runners, but we'll have to
cross that bridge with INFRA if/when it comes up.
-n
On Fri, Feb 6, 2026 at 4:39 PM Charles Connell via dev <de...@hbase.apache.org>
wrote:
> Thanks for your work Nick,
>
> When exploring Github actions myself, I found that they might struggle
> to run the unit tests in the hbase-server module. Any single action
> step is limited to 6 hours runtime, and running these tests can take
> longer than that on the action runner, in my experience. Did you work
> around this somehow?
>
> Charles
>
> On Fri, Feb 6, 2026 at 10:29 AM Nick Dimiduk <nd...@apache.org> wrote:
> >
> > Heya team,
> >
> > We now have static analysis, build, and unit test coverage via GHA
> implemented for PRs against all branches. In the absence of comments,
> you'll find the detailed Yetus table reports in the "Action Summary"
> section of the checks interface. Implementation details are attached to
> HBASE-29838. As before, shout if you find something broken.
> >
> > As of this effort, all of our PR work is being done in GHA. All of the
> Jenkins infrastructure is still present and intact, but I have disabled the
> job in the Jenkins UI [0]. If you're having problems with the new tooling
> and want to restore the Jenkins job, just flip that switch back to enabled.
> >
> > There's more jobs to be converted, I've not filed any additional Jiras
> yet. Please feel encouraged to tackle something if you're feeling so
> inclined.
> >
> > Thanks,
> > Nick
> >
> > [0]: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR
> >
> > On 2026/01/15 14:35:50 Nick Dimiduk wrote:
> > > Hi team,
> > >
> > > I've added GitHub Actions to run Yetus general checks directly on pull
> requests. This has been backported to all active branches (master,
> branch-3, branch-2, branch-2.6, branch-2.5).
> > >
> > > What's changing:
> > > - PRs will now trigger a "Yetus General Check" workflow that runs
> checkstyle, javac, pylint, shellcheck, and other static analysis checks
> > > - Results appear as GitHub status checks and PR comments (same as
> Jenkins)
> > > - Old bot comments from previous runs are automatically hidden to
> reduce noise
> > > - Uses the same scripts, Dockerfile, &c. that we use for Jenkins
> > >
> > > What's NOT changing:
> > > - Jenkins precommit builds continue to run as before
> > > - Your PR workflow remains the same
> > > - Unit/integration tests still run on Jenkins, not GitHub Actions
> > >
> > > JIRAs: HBASE-29787, HBASE-29829
> > >
> > > Give us a shout if anything breaks for you. Also, feel free to pick up
> migrating other jobs, if you want to (I've not filed any subsequent
> tickets). It's a little bit tedious and it seems that you need push rights
> to the origin github repo to introduce a new workflow (I'm not sure about
> editing an existing workflow).
> > >
> > > Thanks,
> > > Nick
> > >
> > > On 2024/10/02 09:22:51 Nick Dimiduk wrote:
> > > > Heya,
> > > >
> > > > I'd like to take the community temperature on migrating our build
> infra
> > > > from the ci-hbase.a.o Jenkins instance to something built on GitHub
> > > > Actions. I have several reasons that justify this proposal.
> > > >
> > > > As some of you may know, our community funding has reduced and we
> will no
> > > > longer be able to sustain the current fleet of build infrastructure.
> So,
> > > > one motivation for this proposal is cost-cutting: I think that we'll
> be
> > > > able to operate at lower costs if we can migrate to a
> provisioned-as-needed
> > > > model of consumption.
> > > >
> > > > My second reason is an optimistic appeal to a larger contributor
> base. I
> > > > suspect that if we can modernize our infrastructure then we will
> increase
> > > > the pool of contributors who might be able to participate in this
> area. I
> > > > believe that GH Actions (and systems like it) is more prevalent in
> the
> > > > industry than Jenkins, which means that more people already have
> experience
> > > > with the platform and more people will feel compelled to offer
> support to
> > > > an OSS project that uses the platform as a means of growing their own
> > > > skillset and as a means of bolstering their CVs.
> > > >
> > > > Dove-tailed into reason two is reason three: I believe that there is
> a
> > > > large community of folks who are developing GitHub Actions on its
> > > > marketplace. We would effectively open ourselves up to more
> off-the-shelf
> > > > offerings and those offerings would be in our hands directly. By
> contrast,
> > > > I don't think there's as much development in Jenkins plugins, and the
> > > > process of adding a new plugin to our Jenkins instance requires
> filing an
> > > > INFRA ticket.
> > > >
> > > > These are my motivations. I'm still not clear on what's possible yet
> for
> > > > ASF projects. I have filed an INFRA ticket, requesting whatever is
> > > > necessary for us to start an experiment. Indeed, I believe that
> there are
> > > > some major limitations on the current implementation provided by the
> ASF,
> > > > and as far as I can tell, only one project with a build footprint
> that
> > > > resembles HBase has pursued this effort. I've catalogued the
> applicable
> > > > information that I've found so far on that issue.
> > > >
> > > > https://issues.apache.org/jira/browse/INFRA-26170
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > >
>
Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI
Posted by Charles Connell via dev <de...@hbase.apache.org>.
Thanks for your work Nick,
When exploring Github actions myself, I found that they might struggle
to run the unit tests in the hbase-server module. Any single action
step is limited to 6 hours runtime, and running these tests can take
longer than that on the action runner, in my experience. Did you work
around this somehow?
Charles
On Fri, Feb 6, 2026 at 10:29 AM Nick Dimiduk <nd...@apache.org> wrote:
>
> Heya team,
>
> We now have static analysis, build, and unit test coverage via GHA implemented for PRs against all branches. In the absence of comments, you'll find the detailed Yetus table reports in the "Action Summary" section of the checks interface. Implementation details are attached to HBASE-29838. As before, shout if you find something broken.
>
> As of this effort, all of our PR work is being done in GHA. All of the Jenkins infrastructure is still present and intact, but I have disabled the job in the Jenkins UI [0]. If you're having problems with the new tooling and want to restore the Jenkins job, just flip that switch back to enabled.
>
> There's more jobs to be converted, I've not filed any additional Jiras yet. Please feel encouraged to tackle something if you're feeling so inclined.
>
> Thanks,
> Nick
>
> [0]: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR
>
> On 2026/01/15 14:35:50 Nick Dimiduk wrote:
> > Hi team,
> >
> > I've added GitHub Actions to run Yetus general checks directly on pull requests. This has been backported to all active branches (master, branch-3, branch-2, branch-2.6, branch-2.5).
> >
> > What's changing:
> > - PRs will now trigger a "Yetus General Check" workflow that runs checkstyle, javac, pylint, shellcheck, and other static analysis checks
> > - Results appear as GitHub status checks and PR comments (same as Jenkins)
> > - Old bot comments from previous runs are automatically hidden to reduce noise
> > - Uses the same scripts, Dockerfile, &c. that we use for Jenkins
> >
> > What's NOT changing:
> > - Jenkins precommit builds continue to run as before
> > - Your PR workflow remains the same
> > - Unit/integration tests still run on Jenkins, not GitHub Actions
> >
> > JIRAs: HBASE-29787, HBASE-29829
> >
> > Give us a shout if anything breaks for you. Also, feel free to pick up migrating other jobs, if you want to (I've not filed any subsequent tickets). It's a little bit tedious and it seems that you need push rights to the origin github repo to introduce a new workflow (I'm not sure about editing an existing workflow).
> >
> > Thanks,
> > Nick
> >
> > On 2024/10/02 09:22:51 Nick Dimiduk wrote:
> > > Heya,
> > >
> > > I'd like to take the community temperature on migrating our build infra
> > > from the ci-hbase.a.o Jenkins instance to something built on GitHub
> > > Actions. I have several reasons that justify this proposal.
> > >
> > > As some of you may know, our community funding has reduced and we will no
> > > longer be able to sustain the current fleet of build infrastructure. So,
> > > one motivation for this proposal is cost-cutting: I think that we'll be
> > > able to operate at lower costs if we can migrate to a provisioned-as-needed
> > > model of consumption.
> > >
> > > My second reason is an optimistic appeal to a larger contributor base. I
> > > suspect that if we can modernize our infrastructure then we will increase
> > > the pool of contributors who might be able to participate in this area. I
> > > believe that GH Actions (and systems like it) is more prevalent in the
> > > industry than Jenkins, which means that more people already have experience
> > > with the platform and more people will feel compelled to offer support to
> > > an OSS project that uses the platform as a means of growing their own
> > > skillset and as a means of bolstering their CVs.
> > >
> > > Dove-tailed into reason two is reason three: I believe that there is a
> > > large community of folks who are developing GitHub Actions on its
> > > marketplace. We would effectively open ourselves up to more off-the-shelf
> > > offerings and those offerings would be in our hands directly. By contrast,
> > > I don't think there's as much development in Jenkins plugins, and the
> > > process of adding a new plugin to our Jenkins instance requires filing an
> > > INFRA ticket.
> > >
> > > These are my motivations. I'm still not clear on what's possible yet for
> > > ASF projects. I have filed an INFRA ticket, requesting whatever is
> > > necessary for us to start an experiment. Indeed, I believe that there are
> > > some major limitations on the current implementation provided by the ASF,
> > > and as far as I can tell, only one project with a build footprint that
> > > resembles HBase has pursued this effort. I've catalogued the applicable
> > > information that I've found so far on that issue.
> > >
> > > https://issues.apache.org/jira/browse/INFRA-26170
> > >
> > > Thanks,
> > > Nick
> > >
> >