Summary of builds meeting Wed 4th Nov 2020

[Gavin McDonald <gm...@apache.org>]

Hi All,

This is extremely late, for which I apologise.
(It's also extremely long, apologies again!)

First I want to thank everyone who attended, we had a great turn out and
I appreciate everyone's time. For me, this was a success and am happy
to now make this a regular thing, having a meeting approximately every
month. Each meeting the content is usually decided in advance on the
wiki page by those that want to attend. I usually have a couple of topics
of my own also.

Next, before the actual summary, I want to give the date of the next
meeting.

Thursday 17th December at 1900 UTC (7PM)

This is a week later than the normal planned schedule but I have
accommodated
for a guest .... At this next meeting we are having someone from Github
come to
answer any questions you might have around Github Actions. (This was a
popular
topic at the last meeting.)

Regarding the meeting platform used. We started off using Jitsi , it was
clear a
few minutes into the meeting that that instance it was running on, was not
big
enough to cope and sure enough it crashed on us. Sander offered up a Google
Meet and after a few minutes the meeting got going again.

I would like for us to give Jitsi another go, basically because it is free,
open source
and no sign up is required in order to join the meeting. This time around
we have
it installed on a bigger instance and the Infra Team have been using it for
the last
few meetings without issue. But there is usually 6/7 people on that call,
not 20, so
a larger audience is still to be tested. I will have a backup url in place
ready to go
should we experience any issues this time around.

Finally, before the summary, if anyone wants to volunteer to also take
notes at
the next meeting as backup for me, and we can compare afterwards before
publishing to this list and the wiki meeting page, that would be awesome.

Talking of meeting notes, I have re-organised the pages a bit to allow for
multiple
meetings. Top page at
https://cwiki.apache.org/confluence/display/INFRA/ASF+Builds+Meetings
contains links to each meeting and also a collated summary of all tasks
listed for all meetings.

Ok, to the meeting summary:-

Meeting started with a slight hiccup, Jitsi crashed on us so we moved to a
Google Meet organised quickly by Sander Striker
<https://cwiki.apache.org/confluence/display/~striker> (thanks Sander!).
Gavin started off by letting everyone know that Docker had upgraded our
DockerHub account and that download limits and image retention limits
would not apply to any project under the 'apache' org namespace on
DockerHub. Gavin also mentioned an upcoming meeting with Github to talk
about usage
stats , container registry etc. (Update on that meeting, GH cant provide
stats as we want them yet, but are able to provide 'snapshots').

Sander talked about his Board level action item, mainly surrounding Infra
documentation on how projects find the CI resources that are available to
them. Sander
later sent an email asking for those on the builds@ list to get involved in
that initiative.

Sander introduced guest Ed Baunton, who together talked about the
advantages of remote execution, Buildgrid and Buildbox, caching of results
of build steps to avoid
repeating steps of a build that have already passed, just send along the
result for the next step, and much more. An action item came of this to
actually build a beta
test service to investigate how this can be of benefit to ASF projects.
Maven folks in particular are a bit skeptical as to whether it could work
for their use case.

Chris Lambertus <https://cwiki.apache.org/confluence/display/~cml> asked
for interest in setting up a K8s cluster of nodes to add to our Jenkins
offering, interest was positive and a couple of people put their hands
up to assist in the creation of this.

Jarek, Vladimir and others were interested in talking about Github Actions,
and most of the rest of the meeting was centered around this topic. Mainly,
how do we know
who is using what resources, where are the bottlenecks, how can we make
better use and be fairer to all. Our account has unlimited minutes, but the
contention is with
the 'runners' of which we have 180. Obviously some visible stats from
Github in this area would help, that is on the list of things that has been
asked for, and hopefully
will be become a reality at some point soon. To ease the load of the
provided runners, some projects want to investigate the possibility of
self-hosted runners. For
open source projects however, this is a potential avenue of attack for
example via PRs. Infra does not endorse self hosted runners at this time -
but also is not blocking
on any project that wants to investigate further. (Perhaps look into
restricting what PRs can trigger a build). Infra will also look into the
security aspects of self-hosted
runners and any possible solutions/workarounds.

Edit: Since the meeting, and the meeting that took place with Github
afterwards, it was suggested that projects should start using a Max
Parallel Setting that is
available for Actions, this would reduce the amount of runners an action
can consume at any one time. Infra suggests to set that at '5'.

During the builds meeting, Gavin pointed people to
https://infra-reports.apache.org/cistats/ ; and suggestions to make this
URL more widely known be adopted.
Whilst unofficial and created originally for Infra internal usage, we can
see its usefulness beyond this and so will get added to docs.

Some good talk was had about Yetus and how it could prove useful for other
projects, beyond what it already does for the Hadoop ecosystem. Allen could
perhaps
talk more about that at a future meeting, maybe a demo of some kind? I
think I recall Allen and Chris talking about Yetus use for Infra, needs
clarification.

This summary is also posted at:-

https://cwiki.apache.org/confluence/display/INFRA/ASF+Builds+Agenda+2020-11-04
I'll have a new wiki page up soon for folks to register their intention to
attend and any
topics of interest they might want to talk about.

Thanks!
-- 

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team

Re: Summary of builds meeting Wed 4th Nov 2020

[Jarek Potiuk <Ja...@polidea.com>]

Hey Gavin?

Any chance for date change please :)?

On Fri, Nov 27, 2020 at 10:03 PM Jarek Potiuk <Ja...@polidea.com>
wrote:

> Hey Gavin,
>
> Would that be possible to move that planned meeting a day or two? The 17th
> of December is very unfortunate for both myself and Tobiasz - we have our
> company Xmas party this day and we have already some activities planned. At
> the same time we are really interested in Github Actions improvements/stats
> and having a guest from GA is something we would not like to miss.
>
> J.
>
> On Wed, Nov 25, 2020 at 10:57 PM Chris Lambertus <cm...@apache.org> wrote:
>
>>
>>
>> > On Nov 25, 2020, at 1:06 PM, Robert Munteanu <ro...@apache.org>
>> wrote:
>> >
>> > Hi,
>> >
>> > On Wed, 2020-11-25 at 20:30 +0100, Gavin McDonald wrote:
>> >> Chris Lambertus <https://cwiki.apache.org/confluence/display/~cml>
>> >> asked
>> >> for interest in setting up a K8s cluster of nodes to add to our
>> >> Jenkins
>> >> offering, interest was positive and a couple of people put their
>> >> hands
>> >> up to assist in the creation of this.
>> >
>> > Can you please detail the k8s initiative. I am not sure if this about
>> > allowing projects to deploy workloads on top of k8s or something else
>> > more tied to Jenkins.
>>
>>
>> The idea is to discuss ways in which projects might use K8s primarily
>> from a CI standpoint, i.e. tied to Jenkins, but if there are other ways
>> such a tool could be used, we’d be happy to hear about them.
>>
>> -Chris
>>
>>
>> >
>> > Thanks,
>> > Robert
>> >
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: Summary of builds meeting Wed 4th Nov 2020

[Jarek Potiuk <Ja...@polidea.com>]

Hey Gavin,

Would that be possible to move that planned meeting a day or two? The 17th
of December is very unfortunate for both myself and Tobiasz - we have our
company Xmas party this day and we have already some activities planned. At
the same time we are really interested in Github Actions improvements/stats
and having a guest from GA is something we would not like to miss.

J.

On Wed, Nov 25, 2020 at 10:57 PM Chris Lambertus <cm...@apache.org> wrote:

>
>
> > On Nov 25, 2020, at 1:06 PM, Robert Munteanu <ro...@apache.org> wrote:
> >
> > Hi,
> >
> > On Wed, 2020-11-25 at 20:30 +0100, Gavin McDonald wrote:
> >> Chris Lambertus <https://cwiki.apache.org/confluence/display/~cml>
> >> asked
> >> for interest in setting up a K8s cluster of nodes to add to our
> >> Jenkins
> >> offering, interest was positive and a couple of people put their
> >> hands
> >> up to assist in the creation of this.
> >
> > Can you please detail the k8s initiative. I am not sure if this about
> > allowing projects to deploy workloads on top of k8s or something else
> > more tied to Jenkins.
>
>
> The idea is to discuss ways in which projects might use K8s primarily from
> a CI standpoint, i.e. tied to Jenkins, but if there are other ways such a
> tool could be used, we’d be happy to hear about them.
>
> -Chris
>
>
> >
> > Thanks,
> > Robert
> >
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: Summary of builds meeting Wed 4th Nov 2020

[Chris Lambertus <cm...@apache.org>]

> On Nov 25, 2020, at 1:06 PM, Robert Munteanu <ro...@apache.org> wrote:
> 
> Hi,
> 
> On Wed, 2020-11-25 at 20:30 +0100, Gavin McDonald wrote:
>> Chris Lambertus <https://cwiki.apache.org/confluence/display/~cml>
>> asked
>> for interest in setting up a K8s cluster of nodes to add to our
>> Jenkins
>> offering, interest was positive and a couple of people put their
>> hands
>> up to assist in the creation of this.
> 
> Can you please detail the k8s initiative. I am not sure if this about
> allowing projects to deploy workloads on top of k8s or something else
> more tied to Jenkins.


The idea is to discuss ways in which projects might use K8s primarily from a CI standpoint, i.e. tied to Jenkins, but if there are other ways such a tool could be used, we’d be happy to hear about them.

-Chris


> 
> Thanks,
> Robert
> 

Re: Summary of builds meeting Wed 4th Nov 2020

[Robert Munteanu <ro...@apache.org>]

Hi,

On Wed, 2020-11-25 at 20:30 +0100, Gavin McDonald wrote:
> Chris Lambertus <https://cwiki.apache.org/confluence/display/~cml>
> asked
> for interest in setting up a K8s cluster of nodes to add to our
> Jenkins
> offering, interest was positive and a couple of people put their
> hands
> up to assist in the creation of this.

Can you please detail the k8s initiative. I am not sure if this about
allowing projects to deploy workloads on top of k8s or something else
more tied to Jenkins.

Thanks,
Robert