You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Otto Fowler <ot...@gmail.com> on 2021/04/19 14:57:19 UTC
Apache Project issues with available GitHub Action resources across
projects
As you can probably imagine, it takes a lot of resources in order to CI for all the Apache projects. Periodically this becomes an issue, as the donated resources from cloud CI providers ( Travis and now GitHub Actions ) end up queuing and delaying builds across Apache projects because of larger projects and their requirements.
The discussions center around a few common themes:
- the CI requirements of large complex projects dominate the Apache Queue
- how those projects can supplement their builds in a way acceptable to ASF INFRA
- how those projects can have per project metering such that a project can pay for hours over the ’norm’
- how to optimize projects
This issue is currently being discussed again on the @builds apache list. I’m sending this over to the Nifi Dev community because Nifi itself has been mentioned as one of the top users of GitHub action resources by some measure. Many of the other projects are actively taking measures to decrease or optimize their usage, and we should probably think about how we can do so as well.
Here is the *current* thread:
https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e <https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E>
Here is the message where 13 projects are listed
https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e <https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E>
There are many other threads regarding GitHub Action limits and resources if you start scrolling back through the archives.
I’m posting this to hopefully kick off some discussions.
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by Mark Bean <ma...@gmail.com>.
Another approach would be to have a time-based build, say no more
frequently than once/day. You would lose granularity to a specific commit,
but reduce compute cycles required by reducing the number of builds being
executed.
Just a thought.
On Mon, Apr 19, 2021 at 2:34 PM David Handermann <ex...@gmail.com>
wrote:
> Chris,,
>
> Thanks for the description of the modular approach. Here is one GitHub
> action that supports conditional execution based on Git changes:
>
> https://github.com/dorny/paths-filter
>
> It would take some effort to implement an approach that covers all the
> bases, but using the Maven also-make and project-list command line options
> should support building both a module and its dependencies. The list of
> changes files can be passed to another action that could determine one or
> more modules to build.
>
> Regards,
> David Handermann
>
> On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson
> <ch...@naimuri.com.invalid> wrote:
>
> > Our approach is the use a `git diff` between source and destination
> > branches when a branch is merged (and the same for on dev branch builds).
> >
> > Each component within the repo then checks for whether any files within
> its
> > directories were changed (bearing in mind a dev branch may consist of
> > multiple commits over time, so don't just look at changes in individual
> > commits).
> >
> > I'm not sure if the mechanics for the existing nifi repo though with its
> > use of:
> >
> > * GutHub Actions (can a git diff be performed at the start and then
> > following actions use the output?)
> >
> > * Maven with sub-modules (can they be optionally built based on such
> > conditions? The GitHub Action could create files depending upon what's
> been
> > changed in order to activate profiles maybe, would that work?)
> >
> > This is broadly how I've implemented a multi-component build previously,
> > but not using Maven sub-modules (each component is typically its own
> docker
> > image and built separately from others).
> >
> >
> > Cheers,
> >
> > Chris Sampson
> >
> > On Mon, 19 Apr 2021, 18:35 Joe Witt, <jo...@gmail.com> wrote:
> >
> > > Chris,
> > >
> > > Yeah that would be very helpful. But do you have any idea how that
> > > might be achieved in this environment?
> > >
> > > Thanks
> > >
> > > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson
> > > <ch...@naimuri.com.invalid> wrote:
> > > >
> > > > Could an approach of building only the updated parts of the repo help
> > to
> > > > reduce build times?
> > > >
> > > > For example, changes to the classes under the AWS bundle (and only
> that
> > > > bundle) would only need those classes to be built and tested.
> > > >
> > > > Where such an approach gets a bit more complex is interdependence
> > between
> > > > parts of the repo. For example, if nifi-api is updated, should all
> > > classes
> > > > be built and tested?
> > > >
> > > > As part of a release, the entire repo could then be built and tested.
> > > >
> > > > This approach might help if the majority of repo changes are to
> > > individual
> > > > NARs rather than wider ranging. I'm also assuming this would be
> > possible
> > > > via GitHub Actions (I have no experience of them, but have
> implemented
> > > this
> > > > kind of approach in a Drone.io mono-repo previously).
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Chris Sampson
> > > >
> > > > On Mon, 19 Apr 2021, 17:16 David Handermann, <
> > exceptionfactory@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > This background is very helpful to keep in mind when evaluating new
> > and
> > > > > updated unit tests. There are definitely some expensive tests that
> > > could
> > > > > be streamlined, but introducing a separate version lifecycle for
> > > framework
> > > > > and extensions seems like it is becoming more necessary. Moving
> to a
> > > Java
> > > > > 11 baseline would also reduce the need to build on multiple
> versions,
> > > but
> > > > > based on other discussions, it sounds like that is not currently
> > > scheduled.
> > > > >
> > > > > I have noticed that Windows builds can run for a longer period of
> > > time, is
> > > > > there a reason that the Windows workflow definition does not have
> the
> > > same
> > > > > timeout setting as the Linux and macOS builds? Introducing one
> would
> > > at
> > > > > least kill off problematic Windows builds.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks for bringing this up. The most clear next step I can
> > envision
> > > > > > at this point is that we break up our core framework from our
> > > > > > extensions. Not obvious how best to break this up but we need
> to.
> > > > > > The build times are insane.
> > > > > >
> > > > > > Joe
> > > > > >
> > > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <
> > ottobackwards@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > As you can probably imagine, it takes a lot of resources in
> order
> > > to CI
> > > > > > for all the Apache projects. Periodically this becomes an issue,
> > as
> > > the
> > > > > > donated resources from cloud CI providers ( Travis and now GitHub
> > > > > Actions )
> > > > > > end up queuing and delaying builds across Apache projects because
> > of
> > > > > larger
> > > > > > projects and their requirements.
> > > > > > >
> > > > > > > The discussions center around a few common themes:
> > > > > > >
> > > > > > > - the CI requirements of large complex projects dominate the
> > Apache
> > > > > Queue
> > > > > > > - how those projects can supplement their builds in a way
> > > acceptable to
> > > > > > ASF INFRA
> > > > > > > - how those projects can have per project metering such that a
> > > project
> > > > > > can pay for hours over the ’norm’
> > > > > > > - how to optimize projects
> > > > > > >
> > > > > > > This issue is currently being discussed again on the @builds
> > apache
> > > > > > list. I’m sending this over to the Nifi Dev community because
> Nifi
> > > > > itself
> > > > > > has been mentioned as one of the top users of GitHub action
> > > resources by
> > > > > > some measure. Many of the other projects are actively taking
> > > measures
> > > > > to
> > > > > > decrease or optimize their usage, and we should probably think
> > about
> > > how
> > > > > we
> > > > > > can do so as well.
> > > > > > >
> > > > > > > Here is the *current* thread:
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> > > > > > <
> > > > > >
> > > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> > > > > > >
> > > > > > >
> > > > > > > Here is the message where 13 projects are listed
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> > > > > > <
> > > > > >
> > > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> > > > > > >
> > > > > > >
> > > > > > > There are many other threads regarding GitHub Action limits and
> > > > > > resources if you start scrolling back through the archives.
> > > > > > >
> > > > > > > I’m posting this to hopefully kick off some discussions.
> > > > > >
> > > > >
> > >
> >
>
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by David Handermann <ex...@gmail.com>.
Chris,,
Thanks for the description of the modular approach. Here is one GitHub
action that supports conditional execution based on Git changes:
https://github.com/dorny/paths-filter
It would take some effort to implement an approach that covers all the
bases, but using the Maven also-make and project-list command line options
should support building both a module and its dependencies. The list of
changes files can be passed to another action that could determine one or
more modules to build.
Regards,
David Handermann
On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
> Our approach is the use a `git diff` between source and destination
> branches when a branch is merged (and the same for on dev branch builds).
>
> Each component within the repo then checks for whether any files within its
> directories were changed (bearing in mind a dev branch may consist of
> multiple commits over time, so don't just look at changes in individual
> commits).
>
> I'm not sure if the mechanics for the existing nifi repo though with its
> use of:
>
> * GutHub Actions (can a git diff be performed at the start and then
> following actions use the output?)
>
> * Maven with sub-modules (can they be optionally built based on such
> conditions? The GitHub Action could create files depending upon what's been
> changed in order to activate profiles maybe, would that work?)
>
> This is broadly how I've implemented a multi-component build previously,
> but not using Maven sub-modules (each component is typically its own docker
> image and built separately from others).
>
>
> Cheers,
>
> Chris Sampson
>
> On Mon, 19 Apr 2021, 18:35 Joe Witt, <jo...@gmail.com> wrote:
>
> > Chris,
> >
> > Yeah that would be very helpful. But do you have any idea how that
> > might be achieved in this environment?
> >
> > Thanks
> >
> > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson
> > <ch...@naimuri.com.invalid> wrote:
> > >
> > > Could an approach of building only the updated parts of the repo help
> to
> > > reduce build times?
> > >
> > > For example, changes to the classes under the AWS bundle (and only that
> > > bundle) would only need those classes to be built and tested.
> > >
> > > Where such an approach gets a bit more complex is interdependence
> between
> > > parts of the repo. For example, if nifi-api is updated, should all
> > classes
> > > be built and tested?
> > >
> > > As part of a release, the entire repo could then be built and tested.
> > >
> > > This approach might help if the majority of repo changes are to
> > individual
> > > NARs rather than wider ranging. I'm also assuming this would be
> possible
> > > via GitHub Actions (I have no experience of them, but have implemented
> > this
> > > kind of approach in a Drone.io mono-repo previously).
> > >
> > >
> > > Cheers,
> > >
> > > Chris Sampson
> > >
> > > On Mon, 19 Apr 2021, 17:16 David Handermann, <
> exceptionfactory@gmail.com
> > >
> > > wrote:
> > >
> > > > This background is very helpful to keep in mind when evaluating new
> and
> > > > updated unit tests. There are definitely some expensive tests that
> > could
> > > > be streamlined, but introducing a separate version lifecycle for
> > framework
> > > > and extensions seems like it is becoming more necessary. Moving to a
> > Java
> > > > 11 baseline would also reduce the need to build on multiple versions,
> > but
> > > > based on other discussions, it sounds like that is not currently
> > scheduled.
> > > >
> > > > I have noticed that Windows builds can run for a longer period of
> > time, is
> > > > there a reason that the Windows workflow definition does not have the
> > same
> > > > timeout setting as the Linux and macOS builds? Introducing one would
> > at
> > > > least kill off problematic Windows builds.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com>
> wrote:
> > > >
> > > > > Thanks for bringing this up. The most clear next step I can
> envision
> > > > > at this point is that we break up our core framework from our
> > > > > extensions. Not obvious how best to break this up but we need to.
> > > > > The build times are insane.
> > > > >
> > > > > Joe
> > > > >
> > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <
> ottobackwards@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > As you can probably imagine, it takes a lot of resources in order
> > to CI
> > > > > for all the Apache projects. Periodically this becomes an issue,
> as
> > the
> > > > > donated resources from cloud CI providers ( Travis and now GitHub
> > > > Actions )
> > > > > end up queuing and delaying builds across Apache projects because
> of
> > > > larger
> > > > > projects and their requirements.
> > > > > >
> > > > > > The discussions center around a few common themes:
> > > > > >
> > > > > > - the CI requirements of large complex projects dominate the
> Apache
> > > > Queue
> > > > > > - how those projects can supplement their builds in a way
> > acceptable to
> > > > > ASF INFRA
> > > > > > - how those projects can have per project metering such that a
> > project
> > > > > can pay for hours over the ’norm’
> > > > > > - how to optimize projects
> > > > > >
> > > > > > This issue is currently being discussed again on the @builds
> apache
> > > > > list. I’m sending this over to the Nifi Dev community because Nifi
> > > > itself
> > > > > has been mentioned as one of the top users of GitHub action
> > resources by
> > > > > some measure. Many of the other projects are actively taking
> > measures
> > > > to
> > > > > decrease or optimize their usage, and we should probably think
> about
> > how
> > > > we
> > > > > can do so as well.
> > > > > >
> > > > > > Here is the *current* thread:
> > > > > >
> > > > >
> > > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> > > > > <
> > > > >
> > > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> > > > > >
> > > > > >
> > > > > > Here is the message where 13 projects are listed
> > > > > >
> > > > >
> > > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> > > > > <
> > > > >
> > > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> > > > > >
> > > > > >
> > > > > > There are many other threads regarding GitHub Action limits and
> > > > > resources if you start scrolling back through the archives.
> > > > > >
> > > > > > I’m posting this to hopefully kick off some discussions.
> > > > >
> > > >
> >
>
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Our approach is the use a `git diff` between source and destination
branches when a branch is merged (and the same for on dev branch builds).
Each component within the repo then checks for whether any files within its
directories were changed (bearing in mind a dev branch may consist of
multiple commits over time, so don't just look at changes in individual
commits).
I'm not sure if the mechanics for the existing nifi repo though with its
use of:
* GutHub Actions (can a git diff be performed at the start and then
following actions use the output?)
* Maven with sub-modules (can they be optionally built based on such
conditions? The GitHub Action could create files depending upon what's been
changed in order to activate profiles maybe, would that work?)
This is broadly how I've implemented a multi-component build previously,
but not using Maven sub-modules (each component is typically its own docker
image and built separately from others).
Cheers,
Chris Sampson
On Mon, 19 Apr 2021, 18:35 Joe Witt, <jo...@gmail.com> wrote:
> Chris,
>
> Yeah that would be very helpful. But do you have any idea how that
> might be achieved in this environment?
>
> Thanks
>
> On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson
> <ch...@naimuri.com.invalid> wrote:
> >
> > Could an approach of building only the updated parts of the repo help to
> > reduce build times?
> >
> > For example, changes to the classes under the AWS bundle (and only that
> > bundle) would only need those classes to be built and tested.
> >
> > Where such an approach gets a bit more complex is interdependence between
> > parts of the repo. For example, if nifi-api is updated, should all
> classes
> > be built and tested?
> >
> > As part of a release, the entire repo could then be built and tested.
> >
> > This approach might help if the majority of repo changes are to
> individual
> > NARs rather than wider ranging. I'm also assuming this would be possible
> > via GitHub Actions (I have no experience of them, but have implemented
> this
> > kind of approach in a Drone.io mono-repo previously).
> >
> >
> > Cheers,
> >
> > Chris Sampson
> >
> > On Mon, 19 Apr 2021, 17:16 David Handermann, <exceptionfactory@gmail.com
> >
> > wrote:
> >
> > > This background is very helpful to keep in mind when evaluating new and
> > > updated unit tests. There are definitely some expensive tests that
> could
> > > be streamlined, but introducing a separate version lifecycle for
> framework
> > > and extensions seems like it is becoming more necessary. Moving to a
> Java
> > > 11 baseline would also reduce the need to build on multiple versions,
> but
> > > based on other discussions, it sounds like that is not currently
> scheduled.
> > >
> > > I have noticed that Windows builds can run for a longer period of
> time, is
> > > there a reason that the Windows workflow definition does not have the
> same
> > > timeout setting as the Linux and macOS builds? Introducing one would
> at
> > > least kill off problematic Windows builds.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Thanks for bringing this up. The most clear next step I can envision
> > > > at this point is that we break up our core framework from our
> > > > extensions. Not obvious how best to break this up but we need to.
> > > > The build times are insane.
> > > >
> > > > Joe
> > > >
> > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <ottobackwards@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > As you can probably imagine, it takes a lot of resources in order
> to CI
> > > > for all the Apache projects. Periodically this becomes an issue, as
> the
> > > > donated resources from cloud CI providers ( Travis and now GitHub
> > > Actions )
> > > > end up queuing and delaying builds across Apache projects because of
> > > larger
> > > > projects and their requirements.
> > > > >
> > > > > The discussions center around a few common themes:
> > > > >
> > > > > - the CI requirements of large complex projects dominate the Apache
> > > Queue
> > > > > - how those projects can supplement their builds in a way
> acceptable to
> > > > ASF INFRA
> > > > > - how those projects can have per project metering such that a
> project
> > > > can pay for hours over the ’norm’
> > > > > - how to optimize projects
> > > > >
> > > > > This issue is currently being discussed again on the @builds apache
> > > > list. I’m sending this over to the Nifi Dev community because Nifi
> > > itself
> > > > has been mentioned as one of the top users of GitHub action
> resources by
> > > > some measure. Many of the other projects are actively taking
> measures
> > > to
> > > > decrease or optimize their usage, and we should probably think about
> how
> > > we
> > > > can do so as well.
> > > > >
> > > > > Here is the *current* thread:
> > > > >
> > > >
> > >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> > > > <
> > > >
> > >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> > > > >
> > > > >
> > > > > Here is the message where 13 projects are listed
> > > > >
> > > >
> > >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> > > > <
> > > >
> > >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> > > > >
> > > > >
> > > > > There are many other threads regarding GitHub Action limits and
> > > > resources if you start scrolling back through the archives.
> > > > >
> > > > > I’m posting this to hopefully kick off some discussions.
> > > >
> > >
>
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by Joe Witt <jo...@gmail.com>.
Chris,
Yeah that would be very helpful. But do you have any idea how that
might be achieved in this environment?
Thanks
On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
>
> Could an approach of building only the updated parts of the repo help to
> reduce build times?
>
> For example, changes to the classes under the AWS bundle (and only that
> bundle) would only need those classes to be built and tested.
>
> Where such an approach gets a bit more complex is interdependence between
> parts of the repo. For example, if nifi-api is updated, should all classes
> be built and tested?
>
> As part of a release, the entire repo could then be built and tested.
>
> This approach might help if the majority of repo changes are to individual
> NARs rather than wider ranging. I'm also assuming this would be possible
> via GitHub Actions (I have no experience of them, but have implemented this
> kind of approach in a Drone.io mono-repo previously).
>
>
> Cheers,
>
> Chris Sampson
>
> On Mon, 19 Apr 2021, 17:16 David Handermann, <ex...@gmail.com>
> wrote:
>
> > This background is very helpful to keep in mind when evaluating new and
> > updated unit tests. There are definitely some expensive tests that could
> > be streamlined, but introducing a separate version lifecycle for framework
> > and extensions seems like it is becoming more necessary. Moving to a Java
> > 11 baseline would also reduce the need to build on multiple versions, but
> > based on other discussions, it sounds like that is not currently scheduled.
> >
> > I have noticed that Windows builds can run for a longer period of time, is
> > there a reason that the Windows workflow definition does not have the same
> > timeout setting as the Linux and macOS builds? Introducing one would at
> > least kill off problematic Windows builds.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Thanks for bringing this up. The most clear next step I can envision
> > > at this point is that we break up our core framework from our
> > > extensions. Not obvious how best to break this up but we need to.
> > > The build times are insane.
> > >
> > > Joe
> > >
> > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > > >
> > > > As you can probably imagine, it takes a lot of resources in order to CI
> > > for all the Apache projects. Periodically this becomes an issue, as the
> > > donated resources from cloud CI providers ( Travis and now GitHub
> > Actions )
> > > end up queuing and delaying builds across Apache projects because of
> > larger
> > > projects and their requirements.
> > > >
> > > > The discussions center around a few common themes:
> > > >
> > > > - the CI requirements of large complex projects dominate the Apache
> > Queue
> > > > - how those projects can supplement their builds in a way acceptable to
> > > ASF INFRA
> > > > - how those projects can have per project metering such that a project
> > > can pay for hours over the ’norm’
> > > > - how to optimize projects
> > > >
> > > > This issue is currently being discussed again on the @builds apache
> > > list. I’m sending this over to the Nifi Dev community because Nifi
> > itself
> > > has been mentioned as one of the top users of GitHub action resources by
> > > some measure. Many of the other projects are actively taking measures
> > to
> > > decrease or optimize their usage, and we should probably think about how
> > we
> > > can do so as well.
> > > >
> > > > Here is the *current* thread:
> > > >
> > >
> > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> > > <
> > >
> > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> > > >
> > > >
> > > > Here is the message where 13 projects are listed
> > > >
> > >
> > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> > > <
> > >
> > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> > > >
> > > >
> > > > There are many other threads regarding GitHub Action limits and
> > > resources if you start scrolling back through the archives.
> > > >
> > > > I’m posting this to hopefully kick off some discussions.
> > >
> >
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Could an approach of building only the updated parts of the repo help to
reduce build times?
For example, changes to the classes under the AWS bundle (and only that
bundle) would only need those classes to be built and tested.
Where such an approach gets a bit more complex is interdependence between
parts of the repo. For example, if nifi-api is updated, should all classes
be built and tested?
As part of a release, the entire repo could then be built and tested.
This approach might help if the majority of repo changes are to individual
NARs rather than wider ranging. I'm also assuming this would be possible
via GitHub Actions (I have no experience of them, but have implemented this
kind of approach in a Drone.io mono-repo previously).
Cheers,
Chris Sampson
On Mon, 19 Apr 2021, 17:16 David Handermann, <ex...@gmail.com>
wrote:
> This background is very helpful to keep in mind when evaluating new and
> updated unit tests. There are definitely some expensive tests that could
> be streamlined, but introducing a separate version lifecycle for framework
> and extensions seems like it is becoming more necessary. Moving to a Java
> 11 baseline would also reduce the need to build on multiple versions, but
> based on other discussions, it sounds like that is not currently scheduled.
>
> I have noticed that Windows builds can run for a longer period of time, is
> there a reason that the Windows workflow definition does not have the same
> timeout setting as the Linux and macOS builds? Introducing one would at
> least kill off problematic Windows builds.
>
> Regards,
> David Handermann
>
> On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Thanks for bringing this up. The most clear next step I can envision
> > at this point is that we break up our core framework from our
> > extensions. Not obvious how best to break this up but we need to.
> > The build times are insane.
> >
> > Joe
> >
> > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <ot...@gmail.com>
> > wrote:
> > >
> > > As you can probably imagine, it takes a lot of resources in order to CI
> > for all the Apache projects. Periodically this becomes an issue, as the
> > donated resources from cloud CI providers ( Travis and now GitHub
> Actions )
> > end up queuing and delaying builds across Apache projects because of
> larger
> > projects and their requirements.
> > >
> > > The discussions center around a few common themes:
> > >
> > > - the CI requirements of large complex projects dominate the Apache
> Queue
> > > - how those projects can supplement their builds in a way acceptable to
> > ASF INFRA
> > > - how those projects can have per project metering such that a project
> > can pay for hours over the ’norm’
> > > - how to optimize projects
> > >
> > > This issue is currently being discussed again on the @builds apache
> > list. I’m sending this over to the Nifi Dev community because Nifi
> itself
> > has been mentioned as one of the top users of GitHub action resources by
> > some measure. Many of the other projects are actively taking measures
> to
> > decrease or optimize their usage, and we should probably think about how
> we
> > can do so as well.
> > >
> > > Here is the *current* thread:
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> > <
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> > >
> > >
> > > Here is the message where 13 projects are listed
> > >
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> > <
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> > >
> > >
> > > There are many other threads regarding GitHub Action limits and
> > resources if you start scrolling back through the archives.
> > >
> > > I’m posting this to hopefully kick off some discussions.
> >
>
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by David Handermann <ex...@gmail.com>.
This background is very helpful to keep in mind when evaluating new and
updated unit tests. There are definitely some expensive tests that could
be streamlined, but introducing a separate version lifecycle for framework
and extensions seems like it is becoming more necessary. Moving to a Java
11 baseline would also reduce the need to build on multiple versions, but
based on other discussions, it sounds like that is not currently scheduled.
I have noticed that Windows builds can run for a longer period of time, is
there a reason that the Windows workflow definition does not have the same
timeout setting as the Linux and macOS builds? Introducing one would at
least kill off problematic Windows builds.
Regards,
David Handermann
On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <jo...@gmail.com> wrote:
> Thanks for bringing this up. The most clear next step I can envision
> at this point is that we break up our core framework from our
> extensions. Not obvious how best to break this up but we need to.
> The build times are insane.
>
> Joe
>
> On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <ot...@gmail.com>
> wrote:
> >
> > As you can probably imagine, it takes a lot of resources in order to CI
> for all the Apache projects. Periodically this becomes an issue, as the
> donated resources from cloud CI providers ( Travis and now GitHub Actions )
> end up queuing and delaying builds across Apache projects because of larger
> projects and their requirements.
> >
> > The discussions center around a few common themes:
> >
> > - the CI requirements of large complex projects dominate the Apache Queue
> > - how those projects can supplement their builds in a way acceptable to
> ASF INFRA
> > - how those projects can have per project metering such that a project
> can pay for hours over the ’norm’
> > - how to optimize projects
> >
> > This issue is currently being discussed again on the @builds apache
> list. I’m sending this over to the Nifi Dev community because Nifi itself
> has been mentioned as one of the top users of GitHub action resources by
> some measure. Many of the other projects are actively taking measures to
> decrease or optimize their usage, and we should probably think about how we
> can do so as well.
> >
> > Here is the *current* thread:
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e
> <
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E
> >
> >
> > Here is the message where 13 projects are listed
> >
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e
> <
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E
> >
> >
> > There are many other threads regarding GitHub Action limits and
> resources if you start scrolling back through the archives.
> >
> > I’m posting this to hopefully kick off some discussions.
>
Re: Apache Project issues with available GitHub Action resources
across projects
Posted by Joe Witt <jo...@gmail.com>.
Thanks for bringing this up. The most clear next step I can envision
at this point is that we break up our core framework from our
extensions. Not obvious how best to break this up but we need to.
The build times are insane.
Joe
On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <ot...@gmail.com> wrote:
>
> As you can probably imagine, it takes a lot of resources in order to CI for all the Apache projects. Periodically this becomes an issue, as the donated resources from cloud CI providers ( Travis and now GitHub Actions ) end up queuing and delaying builds across Apache projects because of larger projects and their requirements.
>
> The discussions center around a few common themes:
>
> - the CI requirements of large complex projects dominate the Apache Queue
> - how those projects can supplement their builds in a way acceptable to ASF INFRA
> - how those projects can have per project metering such that a project can pay for hours over the ’norm’
> - how to optimize projects
>
> This issue is currently being discussed again on the @builds apache list. I’m sending this over to the Nifi Dev community because Nifi itself has been mentioned as one of the top users of GitHub action resources by some measure. Many of the other projects are actively taking measures to decrease or optimize their usage, and we should probably think about how we can do so as well.
>
> Here is the *current* thread:
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3e <https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=CVrghS7d5hg@mail.gmail.com%3E>
>
> Here is the message where 13 projects are listed
> https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3e <https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3C86d8e65a-4943-cbf1-ec46-3980128b8e97@apache.org%3E>
>
> There are many other threads regarding GitHub Action limits and resources if you start scrolling back through the archives.
>
> I’m posting this to hopefully kick off some discussions.