Status of our CI configs (Jenkins and GitHub on-demand builder)

[Christopher <ct...@apache.org>]

Hey Accumulo Devs,

I added a GitHub Actions workflow that is of `workflow_dispatch` type
in our main repo, so committers can trigger a build on-demand from the
GitHub UI on the Actions tab of our repo. The workflow is
parameterized, so you can specify the Maven flags and options you want
for the run, such as running a build with `-Psunny`, or all ITs, or a
specific IT, or some other build flags. I created separate parameters
to group some common options, but that's just to make it more
user-friendly... all options go on the Maven command-line in the end.

I'm not actually sure if we can successfully get all the ITs to pass
in GitHub Actions within the 6 hour maximum job time allowed by
GitHub, on their builders, though. So, we may have to do further
improvements to our ITs to make this even more useful.

I haven't tried yet, but we may also be able to add a parameter to
specify the refspec to build against, so we could build a PR with the
full ITs or specific tests by specifying its refspec
(refs/pull/<#>/{head,merge}). We can add that later, if necessary.

In any case, I hope you find it useful to have this manual trigger
place. It's just another way to build and test our code. And, if you
want to run it in your fork to test your branch before submitting a
PR, you can simply enable GitHub Actions in your fork, and make sure
your branches are up-to-date with the latest from their corresponding
base branch in the main repo (either 1.10 or main) that has the
workflow config file.

Another way to build our code is in the new Jenkins.

In case you haven't been paying attention or missed the notices,
builds.apache.org is being shut down. I have migrated our jobs to
ci-builds.apache.org, and gave some attention to the configs, such as
adding some Slack notices (which we can shut off if they are
annoying). Like on builds.apache.org, our jobs seem to have serious
performance issues on ci-builds.apache.org. I have started a
conversation with Gavin and other INFRA folks in #asfinfra Slack
channel, because I think it might be related to nproc or nofile
ulimits, and they may be able to do something about that. Unlike
GitHub Actions workflows, Jenkins builds don't seem to be
containerized... and there can be concurrent builds running from other
jobs on the same builder, so running the full ITs is still probably
unworkable because of interference from other jobs, but at least we
can still use Jenkins to publish snapshot builds (which I checked and
that appears to be working fine).

Christopher

RE: [EXTERNAL] Status of our CI configs (Jenkins and GitHub on-demand builder)

[Arvind Shyamsundar <ar...@microsoft.com.INVALID>]

Thank you, Christopher! The workflow_dispatch trigger seems to be new, and I am thrilled that they added it. Really useful.

-----Original Message-----
From: Christopher <ct...@apache.org> 
Sent: Sunday, August 16, 2020 1:01 PM
To: accumulo-dev <de...@accumulo.apache.org>
Subject: [EXTERNAL] Status of our CI configs (Jenkins and GitHub on-demand builder)

Hey Accumulo Devs,

I added a GitHub Actions workflow that is of `workflow_dispatch` type in our main repo, so committers can trigger a build on-demand from the GitHub UI on the Actions tab of our repo. The workflow is parameterized, so you can specify the Maven flags and options you want for the run, such as running a build with `-Psunny`, or all ITs, or a specific IT, or some other build flags. I created separate parameters to group some common options, but that's just to make it more user-friendly... all options go on the Maven command-line in the end.

I'm not actually sure if we can successfully get all the ITs to pass in GitHub Actions within the 6 hour maximum job time allowed by GitHub, on their builders, though. So, we may have to do further improvements to our ITs to make this even more useful.

I haven't tried yet, but we may also be able to add a parameter to specify the refspec to build against, so we could build a PR with the full ITs or specific tests by specifying its refspec (refs/pull/<#>/{head,merge}). We can add that later, if necessary.

In any case, I hope you find it useful to have this manual trigger place. It's just another way to build and test our code. And, if you want to run it in your fork to test your branch before submitting a PR, you can simply enable GitHub Actions in your fork, and make sure your branches are up-to-date with the latest from their corresponding base branch in the main repo (either 1.10 or main) that has the workflow config file.

Another way to build our code is in the new Jenkins.

In case you haven't been paying attention or missed the notices, builds.apache.org is being shut down. I have migrated our jobs to ci-builds.apache.org, and gave some attention to the configs, such as adding some Slack notices (which we can shut off if they are annoying). Like on builds.apache.org, our jobs seem to have serious performance issues on ci-builds.apache.org. I have started a conversation with Gavin and other INFRA folks in #asfinfra Slack channel, because I think it might be related to nproc or nofile ulimits, and they may be able to do something about that. Unlike GitHub Actions workflows, Jenkins builds don't seem to be containerized... and there can be concurrent builds running from other jobs on the same builder, so running the full ITs is still probably unworkable because of interference from other jobs, but at least we can still use Jenkins to publish snapshot builds (which I checked and that appears to be working fine).

Christopher