You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Jarek Potiuk <ja...@potiuk.com> on 2021/02/08 00:44:41 UTC
Re: GA again unreasonably slow (again)
For anyone following this thread - some update from the progress we have in
Airflow on building self-hosted infrastructure for GitHub actions.
Ash from Airflow is really close to finalizing the work on a nice
auto-scaling framework for self-hosted workers, but also we checked what is
the best value for money we can get.
I've run some analysis on the performance and tested my hypothesis (based
on earlier experiences) of significant optimisations we can get.
I've finished my analysis of potential optimizations we can get on our CI
with the Self-Hosted runners that Ash created. I did some performance
testing and (very crude) comparison of "traditional approach" with Local
SSDs 2 CPU instances running the tests with something I already tested
several times on various CI arrangements - running tests with High-Memory
instances (8CPU 64 GB Mem) and running everything (including docker engine)
in "tmpfs" - huge ramdisk.
Seems that 1h 20 minutes of test running can be decreased 8x (!)using this
approach (and parallelising some tests) at the same time decreasing the
cost 2x (!). Yep. You heard right. We can have faster builds this way and
pay less for that. Seems that we will be able to decrease the time to run
all tests for one combination to 10 minutes from 1h20 minutes.
This is possible because Ash and his team did a great job on setting up
auto-scaling EC2 instance runners on our Amazon EC2 account (we have
credits from Amazon to run those jobs - also Astronomer offered donation to
keep it running ). Seems that by utilizing it we can not only pay less but
also get much faster builds.
If you are interested - my document is here. Open for comments - happy to
add you as editors if you want (just send me your gmail address in priv).
It is rather crude, I had no time to put a bit more effort into it due to
some significant changes in my company, but it should be easy to compare
the values and see the actual improvements we can get. There are likely a
few shortcuts there and some of the numbers are "back-of-the-envelope" and
we are going to validate them even more when we implement all the
optimisations, but the conclusions should be pretty sound.
https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
J.
On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> We should be able to make an efficient query via GraphQL API right? I found
>> the REST API for actions to be a little underwhelming.
>
>
> That was the first thing I checked when we started looking at the stats.
> Unfortunately last time that I checked (and I even opened an issue for
> that to
> Github support) there was not a Github Actions GraphQL API.
>
> I got a GH support answer "Yeah we know GH API does not have
> GraphQL support yet, sorry". I think it has not changed since.
>
>
> We have tried to make our builds faster with more caching but it's not easy
>> since it's an embedded systems project we need to target a lot of
>> configurations and most changes impact all builds.
>>
>
> Indeed, I know how much of my time was spent on optimising Airflow GH
> usage.
> I think we eventually decreased the usage 10x or more. But it never
> helped, for a
> long as currently anyone even accidentally could block all the slots in
> almost no
> time at all. We have no organisation-wide way to block this and this is
> the problem.
>
> Right now I could:
> a) mine cryptocurrency using PRs to any Apache project
> b) block the queue for everone
>
> I do not have to be even an Apache committer to do that. It's enough if
> just open one PR
> which is well crafted and spins of 180 jobs that run for 6 hours. It's
> super-flawed.
>
>
>>
>> We too would like to would like to take advantage of our own runners but
>> more for the ability to do Hardware In the Loop testing but have avoided
>> it
>> for the reasons already mentioned.
>>
>
> Self-hosted runner for now seems to be the only "Reasonable" option but
> the security
> issues with the current runner are not allowing us to do it.
>
>>
>> --Brennan
>>
>
>
> --
> +48 660 796 129
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
The preliminary description (Still lacking some recent changes and details)
is here:
https://cwiki.apache.org/confluence/display/INFRA/Self-hosted+GitHub+runners
and you can grab Ash as he mentioned in the comments if you want to get
some more details on it .
On Mon, Feb 8, 2021 at 11:01 PM Chris Lambertus <cm...@apache.org> wrote:
>
>
> > On Feb 8, 2021, at 1:51 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > This uses https://github.com/actions/runner/pull/783 to not have
> > un-trusted users run code (security is based on the actors of the commit
> -
> > commiter’s PRs and direct pushes are allowed to run builds on
> self-hosted
> > runners) on our hosts, and then a combination of a Github Application,
> AWS
> > Lambda and an AWS Auto-Scaling Group
>
>
> I’d be interested in additional details on how you’ve implemented Lambda
> and AWS Auto-scaling for this.
>
> -Chris
>
>
> >
> > pon., 8 lut 2021, 09:58 użytkownik Antoine Pitrou <an...@python.org>
> > napisał:
> >
> >>
> >> Hi Jarek,
> >>
> >> Thank you for the document. Could you tell us more about the "custom
> >> security layer" that you implemented?
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> Le 08/02/2021 à 01:44, Jarek Potiuk a écrit :
> >>> For anyone following this thread - some update from the progress we
> have
> >> in
> >>> Airflow on building self-hosted infrastructure for GitHub actions.
> >>>
> >>> Ash from Airflow is really close to finalizing the work on a nice
> >>> auto-scaling framework for self-hosted workers, but also we checked
> what
> >> is
> >>> the best value for money we can get.
> >>>
> >>> I've run some analysis on the performance and tested my hypothesis
> (based
> >>> on earlier experiences) of significant optimisations we can get.
> >>>
> >>> I've finished my analysis of potential optimizations we can get on our
> CI
> >>> with the Self-Hosted runners that Ash created. I did some performance
> >>> testing and (very crude) comparison of "traditional approach" with
> Local
> >>> SSDs 2 CPU instances running the tests with something I already tested
> >>> several times on various CI arrangements - running tests with
> High-Memory
> >>> instances (8CPU 64 GB Mem) and running everything (including docker
> >> engine)
> >>> in "tmpfs" - huge ramdisk.
> >>> Seems that 1h 20 minutes of test running can be decreased 8x (!)using
> >> this
> >>> approach (and parallelising some tests) at the same time decreasing the
> >>> cost 2x (!). Yep. You heard right. We can have faster builds this way
> and
> >>> pay less for that. Seems that we will be able to decrease the time to
> run
> >>> all tests for one combination to 10 minutes from 1h20 minutes.
> >>> This is possible because Ash and his team did a great job on setting up
> >>> auto-scaling EC2 instance runners on our Amazon EC2 account (we have
> >>> credits from Amazon to run those jobs - also Astronomer offered
> donation
> >> to
> >>> keep it running ). Seems that by utilizing it we can not only pay less
> >> but
> >>> also get much faster builds.
> >>>
> >>> If you are interested - my document is here. Open for comments - happy
> to
> >>> add you as editors if you want (just send me your gmail address in
> priv).
> >>> It is rather crude, I had no time to put a bit more effort into it due
> to
> >>> some significant changes in my company, but it should be easy to
> compare
> >>> the values and see the actual improvements we can get. There are
> likely a
> >>> few shortcuts there and some of the numbers are "back-of-the-envelope"
> >> and
> >>> we are going to validate them even more when we implement all the
> >>> optimisations, but the conclusions should be pretty sound.
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
> >>>
> >>> J.
> >>>
> >>>
> >>> On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>>
> >>>> We should be able to make an efficient query via GraphQL API right? I
> >> found
> >>>>> the REST API for actions to be a little underwhelming.
> >>>>
> >>>>
> >>>> That was the first thing I checked when we started looking at the
> stats.
> >>>> Unfortunately last time that I checked (and I even opened an issue for
> >>>> that to
> >>>> Github support) there was not a Github Actions GraphQL API.
> >>>>
> >>>> I got a GH support answer "Yeah we know GH API does not have
> >>>> GraphQL support yet, sorry". I think it has not changed since.
> >>>>
> >>>>
> >>>> We have tried to make our builds faster with more caching but it's not
> >> easy
> >>>>> since it's an embedded systems project we need to target a lot of
> >>>>> configurations and most changes impact all builds.
> >>>>>
> >>>>
> >>>> Indeed, I know how much of my time was spent on optimising Airflow GH
> >>>> usage.
> >>>> I think we eventually decreased the usage 10x or more. But it never
> >>>> helped, for a
> >>>> long as currently anyone even accidentally could block all the slots
> in
> >>>> almost no
> >>>> time at all. We have no organisation-wide way to block this and this
> is
> >>>> the problem.
> >>>>
> >>>> Right now I could:
> >>>> a) mine cryptocurrency using PRs to any Apache project
> >>>> b) block the queue for everone
> >>>>
> >>>> I do not have to be even an Apache committer to do that. It's enough
> if
> >>>> just open one PR
> >>>> which is well crafted and spins of 180 jobs that run for 6 hours. It's
> >>>> super-flawed.
> >>>>
> >>>>
> >>>>>
> >>>>> We too would like to would like to take advantage of our own runners
> >> but
> >>>>> more for the ability to do Hardware In the Loop testing but have
> >> avoided
> >>>>> it
> >>>>> for the reasons already mentioned.
> >>>>>
> >>>>
> >>>> Self-hosted runner for now seems to be the only "Reasonable" option
> but
> >>>> the security
> >>>> issues with the current runner are not allowing us to do it.
> >>>>
> >>>>>
> >>>>> --Brennan
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> +48 660 796 129
> >>>>
> >>>
> >>>
> >>
>
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Chris Lambertus <cm...@apache.org>.
> On Feb 8, 2021, at 1:51 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> This uses https://github.com/actions/runner/pull/783 to not have
> un-trusted users run code (security is based on the actors of the commit -
> commiter’s PRs and direct pushes are allowed to run builds on self-hosted
> runners) on our hosts, and then a combination of a Github Application, AWS
> Lambda and an AWS Auto-Scaling Group
I’d be interested in additional details on how you’ve implemented Lambda and AWS Auto-scaling for this.
-Chris
>
> pon., 8 lut 2021, 09:58 użytkownik Antoine Pitrou <an...@python.org>
> napisał:
>
>>
>> Hi Jarek,
>>
>> Thank you for the document. Could you tell us more about the "custom
>> security layer" that you implemented?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 08/02/2021 à 01:44, Jarek Potiuk a écrit :
>>> For anyone following this thread - some update from the progress we have
>> in
>>> Airflow on building self-hosted infrastructure for GitHub actions.
>>>
>>> Ash from Airflow is really close to finalizing the work on a nice
>>> auto-scaling framework for self-hosted workers, but also we checked what
>> is
>>> the best value for money we can get.
>>>
>>> I've run some analysis on the performance and tested my hypothesis (based
>>> on earlier experiences) of significant optimisations we can get.
>>>
>>> I've finished my analysis of potential optimizations we can get on our CI
>>> with the Self-Hosted runners that Ash created. I did some performance
>>> testing and (very crude) comparison of "traditional approach" with Local
>>> SSDs 2 CPU instances running the tests with something I already tested
>>> several times on various CI arrangements - running tests with High-Memory
>>> instances (8CPU 64 GB Mem) and running everything (including docker
>> engine)
>>> in "tmpfs" - huge ramdisk.
>>> Seems that 1h 20 minutes of test running can be decreased 8x (!)using
>> this
>>> approach (and parallelising some tests) at the same time decreasing the
>>> cost 2x (!). Yep. You heard right. We can have faster builds this way and
>>> pay less for that. Seems that we will be able to decrease the time to run
>>> all tests for one combination to 10 minutes from 1h20 minutes.
>>> This is possible because Ash and his team did a great job on setting up
>>> auto-scaling EC2 instance runners on our Amazon EC2 account (we have
>>> credits from Amazon to run those jobs - also Astronomer offered donation
>> to
>>> keep it running ). Seems that by utilizing it we can not only pay less
>> but
>>> also get much faster builds.
>>>
>>> If you are interested - my document is here. Open for comments - happy to
>>> add you as editors if you want (just send me your gmail address in priv).
>>> It is rather crude, I had no time to put a bit more effort into it due to
>>> some significant changes in my company, but it should be easy to compare
>>> the values and see the actual improvements we can get. There are likely a
>>> few shortcuts there and some of the numbers are "back-of-the-envelope"
>> and
>>> we are going to validate them even more when we implement all the
>>> optimisations, but the conclusions should be pretty sound.
>>>
>>>
>> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
>>>
>>> J.
>>>
>>>
>>> On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>>
>>>> We should be able to make an efficient query via GraphQL API right? I
>> found
>>>>> the REST API for actions to be a little underwhelming.
>>>>
>>>>
>>>> That was the first thing I checked when we started looking at the stats.
>>>> Unfortunately last time that I checked (and I even opened an issue for
>>>> that to
>>>> Github support) there was not a Github Actions GraphQL API.
>>>>
>>>> I got a GH support answer "Yeah we know GH API does not have
>>>> GraphQL support yet, sorry". I think it has not changed since.
>>>>
>>>>
>>>> We have tried to make our builds faster with more caching but it's not
>> easy
>>>>> since it's an embedded systems project we need to target a lot of
>>>>> configurations and most changes impact all builds.
>>>>>
>>>>
>>>> Indeed, I know how much of my time was spent on optimising Airflow GH
>>>> usage.
>>>> I think we eventually decreased the usage 10x or more. But it never
>>>> helped, for a
>>>> long as currently anyone even accidentally could block all the slots in
>>>> almost no
>>>> time at all. We have no organisation-wide way to block this and this is
>>>> the problem.
>>>>
>>>> Right now I could:
>>>> a) mine cryptocurrency using PRs to any Apache project
>>>> b) block the queue for everone
>>>>
>>>> I do not have to be even an Apache committer to do that. It's enough if
>>>> just open one PR
>>>> which is well crafted and spins of 180 jobs that run for 6 hours. It's
>>>> super-flawed.
>>>>
>>>>
>>>>>
>>>>> We too would like to would like to take advantage of our own runners
>> but
>>>>> more for the ability to do Hardware In the Loop testing but have
>> avoided
>>>>> it
>>>>> for the reasons already mentioned.
>>>>>
>>>>
>>>> Self-hosted runner for now seems to be the only "Reasonable" option but
>>>> the security
>>>> issues with the current runner are not allowing us to do it.
>>>>
>>>>>
>>>>> --Brennan
>>>>>
>>>>
>>>>
>>>> --
>>>> +48 660 796 129
>>>>
>>>
>>>
>>
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
This uses https://github.com/actions/runner/pull/783 to not have
un-trusted users run code (security is based on the actors of the commit -
commiter’s PRs and direct pushes are allowed to run builds on self-hosted
runners) on our hosts, and then a combination of a Github Application, AWS
Lambda and an AWS Auto-Scaling Group
pon., 8 lut 2021, 09:58 użytkownik Antoine Pitrou <an...@python.org>
napisał:
>
> Hi Jarek,
>
> Thank you for the document. Could you tell us more about the "custom
> security layer" that you implemented?
>
> Regards
>
> Antoine.
>
>
> Le 08/02/2021 à 01:44, Jarek Potiuk a écrit :
> > For anyone following this thread - some update from the progress we have
> in
> > Airflow on building self-hosted infrastructure for GitHub actions.
> >
> > Ash from Airflow is really close to finalizing the work on a nice
> > auto-scaling framework for self-hosted workers, but also we checked what
> is
> > the best value for money we can get.
> >
> > I've run some analysis on the performance and tested my hypothesis (based
> > on earlier experiences) of significant optimisations we can get.
> >
> > I've finished my analysis of potential optimizations we can get on our CI
> > with the Self-Hosted runners that Ash created. I did some performance
> > testing and (very crude) comparison of "traditional approach" with Local
> > SSDs 2 CPU instances running the tests with something I already tested
> > several times on various CI arrangements - running tests with High-Memory
> > instances (8CPU 64 GB Mem) and running everything (including docker
> engine)
> > in "tmpfs" - huge ramdisk.
> > Seems that 1h 20 minutes of test running can be decreased 8x (!)using
> this
> > approach (and parallelising some tests) at the same time decreasing the
> > cost 2x (!). Yep. You heard right. We can have faster builds this way and
> > pay less for that. Seems that we will be able to decrease the time to run
> > all tests for one combination to 10 minutes from 1h20 minutes.
> > This is possible because Ash and his team did a great job on setting up
> > auto-scaling EC2 instance runners on our Amazon EC2 account (we have
> > credits from Amazon to run those jobs - also Astronomer offered donation
> to
> > keep it running ). Seems that by utilizing it we can not only pay less
> but
> > also get much faster builds.
> >
> > If you are interested - my document is here. Open for comments - happy to
> > add you as editors if you want (just send me your gmail address in priv).
> > It is rather crude, I had no time to put a bit more effort into it due to
> > some significant changes in my company, but it should be easy to compare
> > the values and see the actual improvements we can get. There are likely a
> > few shortcuts there and some of the numbers are "back-of-the-envelope"
> and
> > we are going to validate them even more when we implement all the
> > optimisations, but the conclusions should be pretty sound.
> >
> >
> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
> >
> > J.
> >
> >
> > On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >>
> >> We should be able to make an efficient query via GraphQL API right? I
> found
> >>> the REST API for actions to be a little underwhelming.
> >>
> >>
> >> That was the first thing I checked when we started looking at the stats.
> >> Unfortunately last time that I checked (and I even opened an issue for
> >> that to
> >> Github support) there was not a Github Actions GraphQL API.
> >>
> >> I got a GH support answer "Yeah we know GH API does not have
> >> GraphQL support yet, sorry". I think it has not changed since.
> >>
> >>
> >> We have tried to make our builds faster with more caching but it's not
> easy
> >>> since it's an embedded systems project we need to target a lot of
> >>> configurations and most changes impact all builds.
> >>>
> >>
> >> Indeed, I know how much of my time was spent on optimising Airflow GH
> >> usage.
> >> I think we eventually decreased the usage 10x or more. But it never
> >> helped, for a
> >> long as currently anyone even accidentally could block all the slots in
> >> almost no
> >> time at all. We have no organisation-wide way to block this and this is
> >> the problem.
> >>
> >> Right now I could:
> >> a) mine cryptocurrency using PRs to any Apache project
> >> b) block the queue for everone
> >>
> >> I do not have to be even an Apache committer to do that. It's enough if
> >> just open one PR
> >> which is well crafted and spins of 180 jobs that run for 6 hours. It's
> >> super-flawed.
> >>
> >>
> >>>
> >>> We too would like to would like to take advantage of our own runners
> but
> >>> more for the ability to do Hardware In the Loop testing but have
> avoided
> >>> it
> >>> for the reasons already mentioned.
> >>>
> >>
> >> Self-hosted runner for now seems to be the only "Reasonable" option but
> >> the security
> >> issues with the current runner are not allowing us to do it.
> >>
> >>>
> >>> --Brennan
> >>>
> >>
> >>
> >> --
> >> +48 660 796 129
> >>
> >
> >
>
Re: GA again unreasonably slow (again)
Posted by Antoine Pitrou <an...@python.org>.
Hi Jarek,
Thank you for the document. Could you tell us more about the "custom
security layer" that you implemented?
Regards
Antoine.
Le 08/02/2021 à 01:44, Jarek Potiuk a écrit :
> For anyone following this thread - some update from the progress we have in
> Airflow on building self-hosted infrastructure for GitHub actions.
>
> Ash from Airflow is really close to finalizing the work on a nice
> auto-scaling framework for self-hosted workers, but also we checked what is
> the best value for money we can get.
>
> I've run some analysis on the performance and tested my hypothesis (based
> on earlier experiences) of significant optimisations we can get.
>
> I've finished my analysis of potential optimizations we can get on our CI
> with the Self-Hosted runners that Ash created. I did some performance
> testing and (very crude) comparison of "traditional approach" with Local
> SSDs 2 CPU instances running the tests with something I already tested
> several times on various CI arrangements - running tests with High-Memory
> instances (8CPU 64 GB Mem) and running everything (including docker engine)
> in "tmpfs" - huge ramdisk.
> Seems that 1h 20 minutes of test running can be decreased 8x (!)using this
> approach (and parallelising some tests) at the same time decreasing the
> cost 2x (!). Yep. You heard right. We can have faster builds this way and
> pay less for that. Seems that we will be able to decrease the time to run
> all tests for one combination to 10 minutes from 1h20 minutes.
> This is possible because Ash and his team did a great job on setting up
> auto-scaling EC2 instance runners on our Amazon EC2 account (we have
> credits from Amazon to run those jobs - also Astronomer offered donation to
> keep it running ). Seems that by utilizing it we can not only pay less but
> also get much faster builds.
>
> If you are interested - my document is here. Open for comments - happy to
> add you as editors if you want (just send me your gmail address in priv).
> It is rather crude, I had no time to put a bit more effort into it due to
> some significant changes in my company, but it should be easy to compare
> the values and see the actual improvements we can get. There are likely a
> few shortcuts there and some of the numbers are "back-of-the-envelope" and
> we are going to validate them even more when we implement all the
> optimisations, but the conclusions should be pretty sound.
>
> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
>
> J.
>
>
> On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>> We should be able to make an efficient query via GraphQL API right? I found
>>> the REST API for actions to be a little underwhelming.
>>
>>
>> That was the first thing I checked when we started looking at the stats.
>> Unfortunately last time that I checked (and I even opened an issue for
>> that to
>> Github support) there was not a Github Actions GraphQL API.
>>
>> I got a GH support answer "Yeah we know GH API does not have
>> GraphQL support yet, sorry". I think it has not changed since.
>>
>>
>> We have tried to make our builds faster with more caching but it's not easy
>>> since it's an embedded systems project we need to target a lot of
>>> configurations and most changes impact all builds.
>>>
>>
>> Indeed, I know how much of my time was spent on optimising Airflow GH
>> usage.
>> I think we eventually decreased the usage 10x or more. But it never
>> helped, for a
>> long as currently anyone even accidentally could block all the slots in
>> almost no
>> time at all. We have no organisation-wide way to block this and this is
>> the problem.
>>
>> Right now I could:
>> a) mine cryptocurrency using PRs to any Apache project
>> b) block the queue for everone
>>
>> I do not have to be even an Apache committer to do that. It's enough if
>> just open one PR
>> which is well crafted and spins of 180 jobs that run for 6 hours. It's
>> super-flawed.
>>
>>
>>>
>>> We too would like to would like to take advantage of our own runners but
>>> more for the ability to do Hardware In the Loop testing but have avoided
>>> it
>>> for the reasons already mentioned.
>>>
>>
>> Self-hosted runner for now seems to be the only "Reasonable" option but
>> the security
>> issues with the current runner are not allowing us to do it.
>>
>>>
>>> --Brennan
>>>
>>
>>
>> --
>> +48 660 796 129
>>
>
>
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
The reasoning for selective checks here:
https://github.com/apache/airflow/blob/master/PULL_REQUEST_WORKFLOW.rst
(correct link)
On Tue, Feb 9, 2021 at 7:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> | The real hard problem is knowing when a change requires full regression
> and integration testing of all possible platforms.
>
> And here I absolutely agree too. Even more than that - I am a hard
> practitioner of that. This is what we already implemented in Airflow (the
> whole reasoning how and why it is implemented is here:
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#)
> (still we have a few optimisations left). We call it "selective checks".
>
> And this is what I already proposed the Pulsar team to implement too -
> just take a look at chapter 4) in their document:
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#
>
>
> In Airflow it helped a lot at some point. We got ~70% less load on the
> queue - mainly thanks to selective checks.
>
> Unfortunately - with the shared queue of the ASF, it only helped for some
> two weeks - precisely because we were the only obese who have done that,
> so while being gentle to others, we have not got the love back. But - this
> is not a complaint, I think it is a natural thing when you have shared
> resources that you do not pay for. People will not optimise as it is
> huge investment for them (not only the cost of doing it, but also
> increased complexity). It\s been mentioned several times that Airflow's CI
> is over-engineered, but I think it is simply heavily optimized (which
> brings necessary complexity).
>
> Again - there is no way (and it would be even not fair TBH) to enforce
> this optimisation of their processes for the benefits of others, if they
> have no incentives. This is simply consequence of the model of "free shared
> motorway". No matter how hard you try - you will eventually end-up with
> traffic jams.
>
> J.
>
>
> On Tue, Feb 9, 2021 at 6:40 PM Dave Fisher <wa...@comcast.net> wrote:
>
>> The real hard problem is knowing when a change requires full regression
>> and integration testing of all possible platforms.
>>
>> I think projects are allowing lazy engineering if those making changes
>> don’t know the level of testing needed for their changes.
>>
>> Now with easy lightweight branches all being fully tested ....
>>
>> This is my 10,000 meter view.
>>
>> But then I’m old school and on my first job the mainframe printout
>> included how much the run I made was costing my boss in $.
>>
>> Best Regards,
>> Dave
>>
>> Sent from my iPhone
>>
>> > On Feb 9, 2021, at 9:20 AM, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > Absolutely agree Matt. Throwing more hardware at "all of the projects"
>> is
>> > definitely not going to help - I was telling that from the beginning -
>> it
>> > is like building free motorways - the more you build, the more traffic
>> > flows and the traffic jams remain. That's why I think reasonable
>> > self-hosted solution that every project owns (including getting the
>> credits
>> > for that) is the only viable solution IMHO - only then you really start
>> > optimising stuff because you own both - the problem and the solution
>> > (and you do not - uncontrollably) impact other projects.
>> >
>> > We've just opened-up today the self-hosted solution in Airflow -
>> > announcement from Ash here:
>> >
>> https://lists.apache.org/thread.html/r2e398f86479e4cbfca13c22e4499fb0becdbba20dd9d6d47e1ed30bd%40%3Cdev.airflow.apache.org%3E
>> > and we will be working out any "teething problems", once we are past
>> that,
>> >
>> > We are on our way to achieve the goal from the first paragraph - i.e. be
>> > able to control both problem and solution on a per-project basis. And
>> once
>> > we get some learnings - I am sure we will share our solution and
>> findings
>> > more widely with other projects, so that they could apply
>> > similar solutions. This is especially the missing "security piece"
>> which
>> > was a "blocker" so far, but also auto-scaling and tmpfs-optimisation
>> > results (which is a nice side-effect if we can get the 10x improvements
>> in
>> > feedback time eventually (as it seems we can get there).
>> >
>> > We love data @Airflow so we will gather some stats that everyone will be
>> > able to analyse and see how much they can gain from - not only the queue
>> > bottleneck removal but also improving the most important (in my opinion)
>> > metrics for the CI - which is feedback time. I personally think in CI
>> there
>> > are are the only two important metrics: reliability and feedback time.
>> > Nothing else (including cost) matters. But If we get all three improved.
>> > that would be something that we will be happy other projects can also
>> > benefit from.
>> >
>> > J.
>> >
>> >
>> >
>> >> On Tue, Feb 9, 2021 at 3:16 PM Matt Sicker <bo...@gmail.com> wrote:
>> >>
>> >> To be honest, this sounds exactly like the usual CI problem on every
>> >> platform. As your project scales up, CI becomes a Hard Problem. I don’t
>> >> think throwing hardware at it indefinitely works, though your research
>> here
>> >> is finding most of the useful things.
>> >>
>> >>> On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>> The report shows only top contenders. And yes - we know it is flawed -
>> >>> because it shows workflows not jobs (if you read the disclaimers - we
>> >>> simply have not enough API calls quota to get detailed information for
>> >> all
>> >>> projects).
>> >>>
>> >>> So this is anecdotal. I also get no queue when I submit PR at 11 pm.
>> >>> Actually whole Airflow committer team had to switch to the "night
>> shift"
>> >>> because of that. And the most "traffic-heavy" projects - Spark,
>> Pulsar,
>> >>> Superset, Beam, Airflow - I think some of the top "traffic" projects
>> >>> experience the same issues and several hours queue when they run
>> during
>> >> the
>> >>> EMEA day/US morning. And we all together try to help each other (for
>> >>> example I helped yesterday the Pulsar team to implement most
>> aggressive
>> >> way
>> >>> of cancelling their workflows
>> https://github.com/apache/pulsar/pull/9503
>> >>> (you can find pretty good explanation why and how it was implemented
>> this
>> >>> way), also we are working together with the Pulsar team to optimize
>> their
>> >>> workflow - there is a document
>> >>>
>> >>>
>> >>
>> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
>> >>> where several peopel are adding their suggestions (including myself
>> based
>> >>> on Airflow experiences).
>> >>>
>> >>> And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
>> >>> https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you
>> >> have
>> >>> not experienced it, especially that you are the only person committing
>> >>> there. This is hardly representative for other projects that have
>> 100s of
>> >>> committers and 100s of PRs a day. I am not sure if you are aware of
>> >>> that, but those are the most valuable projects for the ASF - as those
>> are
>> >>> the ones that actually build community (Folowing "comunity over code
>> >>> motto). If you have 3 PRs in 3 months and there aare 200 other
>> projects
>> >>> using GA, I think yetus is not going to show up in any meaningful
>> >>> statistics.
>> >>>
>> >>> I am not sure if drawing a conclusion from a project that has 3 PRs
>> in 2
>> >>> months is the best way of drawing conclusions for the overall Apache
>> >>> organisation. I think drawing a conclusion from experiences of 5
>> actually
>> >>> active projects with sometimes even 100 PRs a day is probably better
>> >>> justified (yep - there are such projects).
>> >>> So I would probably agree it has little influence on projects that
>> have
>> >> no
>> >>> traffic. But enormous influence on projects that actually have
>> traffic.
>> >> You
>> >>> have several teams of people scrambling now to somehow manage their
>> CI
>> >> as
>> >>> it is unbearable now. Is this serious ? I'd say so.
>> >>>
>> >>> When you see Airflow backed up, maybe you should try
>> submitting a
>> >>> PR to another project yourself to see what happens.
>> >>>
>> >>> I am already spending a TON of my private time trying to help others
>> in
>> >> the
>> >>> community. I would really appreciate a little help from your side. So
>> >> maybe
>> >>> you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET ->
>> 8pm
>> >>> CET - this is where regularly bottlenecks happen. Please let everyone
>> >> know
>> >>> your findings
>> >>>
>> >>> J,
>> >>>
>> >>>
>> >>> On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
>> >>> <aw...@effectivemachines.com.invalid> wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>>> On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>>>
>> >>>>>> I'm not convinced this is true. I have yet to see any of my PRs for
>> >>>>> "non-big" projects getting queued while Spark, Airflow, others are.
>> >>> Thus
>> >>>>> why I think there are only a handful of projects that are getting
>> >> upset
>> >>>>> about this but the rest of us are like "meh whatever."
>> >>>>>
>> >>>>> Do you have any data on that? Or is it just anecdotal evidence?
>> >>>>
>> >>>> Totally anecdotal. Like when I literally ran a Yetus PR
>> during
>> >>>> the builds meeting as you were complaining about Airflow having an X
>> >> deep
>> >>>> queue. My PR ran fine, no pause.
>> >>>>
>> >>>>> You can see some analysis and actually even charts here:
>> >>>>>
>> >>>
>> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
>> >>>>
>> >>>> Yes, and I don't even see Yetus showing up. I wonder how many
>> >>>> other projects are getting dropped from the dataset....
>> >>>>
>> >>>>> Maybe you have a very tiny "PR traffic" and it is mostly in the time
>> >>> zone
>> >>>>> that is not affected?
>> >>>>
>> >>>> True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
>> >>>> different though) But if it was one big FIFO queue, our PR jobs
>> would
>> >>> also
>> >>>> get queued. They aren't even when I go look at one of the other
>> >> projects
>> >>>> that does have queued jobs.
>> >>>>
>> >>>> When you see Airflow backed up, maybe you should try
>> >> submitting a
>> >>>> PR to another project yourself to see what happens.
>> >>>>
>> >>>> All I'm saying is: right now, that document feels like it is
>> >>>> _greatly_ overstating the problem and now that you point it out,
>> >> clearly
>> >>>> dropping data. It is problem, to be sure, but not all GitHub Actions
>> >>>> projects are suffering. (I wouldn't be surprised if smaller projects
>> >> are
>> >>>> actually fast tracked through the build queue in order to avoid a
>> >> tyranny
>> >>>> of the majority/resource starvation problem... which would be ironic
>> >>> given
>> >>>> how much of an issue that is at the ASF.)
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> +48 660 796 129
>> >>>
>> >>
>> >
>> >
>> > --
>> > +48 660 796 129
>>
>
>
> --
> +48 660 796 129
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
| The real hard problem is knowing when a change requires full regression
and integration testing of all possible platforms.
And here I absolutely agree too. Even more than that - I am a hard
practitioner of that. This is what we already implemented in Airflow (the
whole reasoning how and why it is implemented is here:
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#)
(still we have a few optimisations left). We call it "selective checks".
And this is what I already proposed the Pulsar team to implement too - just
take a look at chapter 4) in their document:
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#
In Airflow it helped a lot at some point. We got ~70% less load on the
queue - mainly thanks to selective checks.
Unfortunately - with the shared queue of the ASF, it only helped for some
two weeks - precisely because we were the only obese who have done that,
so while being gentle to others, we have not got the love back. But - this
is not a complaint, I think it is a natural thing when you have shared
resources that you do not pay for. People will not optimise as it is
huge investment for them (not only the cost of doing it, but also
increased complexity). It\s been mentioned several times that Airflow's CI
is over-engineered, but I think it is simply heavily optimized (which
brings necessary complexity).
Again - there is no way (and it would be even not fair TBH) to enforce this
optimisation of their processes for the benefits of others, if they have no
incentives. This is simply consequence of the model of "free shared
motorway". No matter how hard you try - you will eventually end-up with
traffic jams.
J.
On Tue, Feb 9, 2021 at 6:40 PM Dave Fisher <wa...@comcast.net> wrote:
> The real hard problem is knowing when a change requires full regression
> and integration testing of all possible platforms.
>
> I think projects are allowing lazy engineering if those making changes
> don’t know the level of testing needed for their changes.
>
> Now with easy lightweight branches all being fully tested ....
>
> This is my 10,000 meter view.
>
> But then I’m old school and on my first job the mainframe printout
> included how much the run I made was costing my boss in $.
>
> Best Regards,
> Dave
>
> Sent from my iPhone
>
> > On Feb 9, 2021, at 9:20 AM, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Absolutely agree Matt. Throwing more hardware at "all of the projects"
> is
> > definitely not going to help - I was telling that from the beginning - it
> > is like building free motorways - the more you build, the more traffic
> > flows and the traffic jams remain. That's why I think reasonable
> > self-hosted solution that every project owns (including getting the
> credits
> > for that) is the only viable solution IMHO - only then you really start
> > optimising stuff because you own both - the problem and the solution
> > (and you do not - uncontrollably) impact other projects.
> >
> > We've just opened-up today the self-hosted solution in Airflow -
> > announcement from Ash here:
> >
> https://lists.apache.org/thread.html/r2e398f86479e4cbfca13c22e4499fb0becdbba20dd9d6d47e1ed30bd%40%3Cdev.airflow.apache.org%3E
> > and we will be working out any "teething problems", once we are past
> that,
> >
> > We are on our way to achieve the goal from the first paragraph - i.e. be
> > able to control both problem and solution on a per-project basis. And
> once
> > we get some learnings - I am sure we will share our solution and findings
> > more widely with other projects, so that they could apply
> > similar solutions. This is especially the missing "security piece" which
> > was a "blocker" so far, but also auto-scaling and tmpfs-optimisation
> > results (which is a nice side-effect if we can get the 10x improvements
> in
> > feedback time eventually (as it seems we can get there).
> >
> > We love data @Airflow so we will gather some stats that everyone will be
> > able to analyse and see how much they can gain from - not only the queue
> > bottleneck removal but also improving the most important (in my opinion)
> > metrics for the CI - which is feedback time. I personally think in CI
> there
> > are are the only two important metrics: reliability and feedback time.
> > Nothing else (including cost) matters. But If we get all three improved.
> > that would be something that we will be happy other projects can also
> > benefit from.
> >
> > J.
> >
> >
> >
> >> On Tue, Feb 9, 2021 at 3:16 PM Matt Sicker <bo...@gmail.com> wrote:
> >>
> >> To be honest, this sounds exactly like the usual CI problem on every
> >> platform. As your project scales up, CI becomes a Hard Problem. I don’t
> >> think throwing hardware at it indefinitely works, though your research
> here
> >> is finding most of the useful things.
> >>
> >>> On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> The report shows only top contenders. And yes - we know it is flawed -
> >>> because it shows workflows not jobs (if you read the disclaimers - we
> >>> simply have not enough API calls quota to get detailed information for
> >> all
> >>> projects).
> >>>
> >>> So this is anecdotal. I also get no queue when I submit PR at 11 pm.
> >>> Actually whole Airflow committer team had to switch to the "night
> shift"
> >>> because of that. And the most "traffic-heavy" projects - Spark, Pulsar,
> >>> Superset, Beam, Airflow - I think some of the top "traffic" projects
> >>> experience the same issues and several hours queue when they run during
> >> the
> >>> EMEA day/US morning. And we all together try to help each other (for
> >>> example I helped yesterday the Pulsar team to implement most aggressive
> >> way
> >>> of cancelling their workflows
> https://github.com/apache/pulsar/pull/9503
> >>> (you can find pretty good explanation why and how it was implemented
> this
> >>> way), also we are working together with the Pulsar team to optimize
> their
> >>> workflow - there is a document
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
> >>> where several peopel are adding their suggestions (including myself
> based
> >>> on Airflow experiences).
> >>>
> >>> And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
> >>> https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you
> >> have
> >>> not experienced it, especially that you are the only person committing
> >>> there. This is hardly representative for other projects that have 100s
> of
> >>> committers and 100s of PRs a day. I am not sure if you are aware of
> >>> that, but those are the most valuable projects for the ASF - as those
> are
> >>> the ones that actually build community (Folowing "comunity over code
> >>> motto). If you have 3 PRs in 3 months and there aare 200 other projects
> >>> using GA, I think yetus is not going to show up in any meaningful
> >>> statistics.
> >>>
> >>> I am not sure if drawing a conclusion from a project that has 3 PRs in
> 2
> >>> months is the best way of drawing conclusions for the overall Apache
> >>> organisation. I think drawing a conclusion from experiences of 5
> actually
> >>> active projects with sometimes even 100 PRs a day is probably better
> >>> justified (yep - there are such projects).
> >>> So I would probably agree it has little influence on projects that have
> >> no
> >>> traffic. But enormous influence on projects that actually have traffic.
> >> You
> >>> have several teams of people scrambling now to somehow manage their CI
> >> as
> >>> it is unbearable now. Is this serious ? I'd say so.
> >>>
> >>> When you see Airflow backed up, maybe you should try submitting
> a
> >>> PR to another project yourself to see what happens.
> >>>
> >>> I am already spending a TON of my private time trying to help others in
> >> the
> >>> community. I would really appreciate a little help from your side. So
> >> maybe
> >>> you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET ->
> 8pm
> >>> CET - this is where regularly bottlenecks happen. Please let everyone
> >> know
> >>> your findings
> >>>
> >>> J,
> >>>
> >>>
> >>> On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
> >>> <aw...@effectivemachines.com.invalid> wrote:
> >>>
> >>>>
> >>>>
> >>>>> On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>>
> >>>>>> I'm not convinced this is true. I have yet to see any of my PRs for
> >>>>> "non-big" projects getting queued while Spark, Airflow, others are.
> >>> Thus
> >>>>> why I think there are only a handful of projects that are getting
> >> upset
> >>>>> about this but the rest of us are like "meh whatever."
> >>>>>
> >>>>> Do you have any data on that? Or is it just anecdotal evidence?
> >>>>
> >>>> Totally anecdotal. Like when I literally ran a Yetus PR during
> >>>> the builds meeting as you were complaining about Airflow having an X
> >> deep
> >>>> queue. My PR ran fine, no pause.
> >>>>
> >>>>> You can see some analysis and actually even charts here:
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
> >>>>
> >>>> Yes, and I don't even see Yetus showing up. I wonder how many
> >>>> other projects are getting dropped from the dataset....
> >>>>
> >>>>> Maybe you have a very tiny "PR traffic" and it is mostly in the time
> >>> zone
> >>>>> that is not affected?
> >>>>
> >>>> True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
> >>>> different though) But if it was one big FIFO queue, our PR jobs would
> >>> also
> >>>> get queued. They aren't even when I go look at one of the other
> >> projects
> >>>> that does have queued jobs.
> >>>>
> >>>> When you see Airflow backed up, maybe you should try
> >> submitting a
> >>>> PR to another project yourself to see what happens.
> >>>>
> >>>> All I'm saying is: right now, that document feels like it is
> >>>> _greatly_ overstating the problem and now that you point it out,
> >> clearly
> >>>> dropping data. It is problem, to be sure, but not all GitHub Actions
> >>>> projects are suffering. (I wouldn't be surprised if smaller projects
> >> are
> >>>> actually fast tracked through the build queue in order to avoid a
> >> tyranny
> >>>> of the majority/resource starvation problem... which would be ironic
> >>> given
> >>>> how much of an issue that is at the ASF.)
> >>>
> >>>
> >>>
> >>> --
> >>> +48 660 796 129
> >>>
> >>
> >
> >
> > --
> > +48 660 796 129
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Dave Fisher <wa...@comcast.net>.
The real hard problem is knowing when a change requires full regression and integration testing of all possible platforms.
I think projects are allowing lazy engineering if those making changes don’t know the level of testing needed for their changes.
Now with easy lightweight branches all being fully tested ....
This is my 10,000 meter view.
But then I’m old school and on my first job the mainframe printout included how much the run I made was costing my boss in $.
Best Regards,
Dave
Sent from my iPhone
> On Feb 9, 2021, at 9:20 AM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Absolutely agree Matt. Throwing more hardware at "all of the projects" is
> definitely not going to help - I was telling that from the beginning - it
> is like building free motorways - the more you build, the more traffic
> flows and the traffic jams remain. That's why I think reasonable
> self-hosted solution that every project owns (including getting the credits
> for that) is the only viable solution IMHO - only then you really start
> optimising stuff because you own both - the problem and the solution
> (and you do not - uncontrollably) impact other projects.
>
> We've just opened-up today the self-hosted solution in Airflow -
> announcement from Ash here:
> https://lists.apache.org/thread.html/r2e398f86479e4cbfca13c22e4499fb0becdbba20dd9d6d47e1ed30bd%40%3Cdev.airflow.apache.org%3E
> and we will be working out any "teething problems", once we are past that,
>
> We are on our way to achieve the goal from the first paragraph - i.e. be
> able to control both problem and solution on a per-project basis. And once
> we get some learnings - I am sure we will share our solution and findings
> more widely with other projects, so that they could apply
> similar solutions. This is especially the missing "security piece" which
> was a "blocker" so far, but also auto-scaling and tmpfs-optimisation
> results (which is a nice side-effect if we can get the 10x improvements in
> feedback time eventually (as it seems we can get there).
>
> We love data @Airflow so we will gather some stats that everyone will be
> able to analyse and see how much they can gain from - not only the queue
> bottleneck removal but also improving the most important (in my opinion)
> metrics for the CI - which is feedback time. I personally think in CI there
> are are the only two important metrics: reliability and feedback time.
> Nothing else (including cost) matters. But If we get all three improved.
> that would be something that we will be happy other projects can also
> benefit from.
>
> J.
>
>
>
>> On Tue, Feb 9, 2021 at 3:16 PM Matt Sicker <bo...@gmail.com> wrote:
>>
>> To be honest, this sounds exactly like the usual CI problem on every
>> platform. As your project scales up, CI becomes a Hard Problem. I don’t
>> think throwing hardware at it indefinitely works, though your research here
>> is finding most of the useful things.
>>
>>> On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> The report shows only top contenders. And yes - we know it is flawed -
>>> because it shows workflows not jobs (if you read the disclaimers - we
>>> simply have not enough API calls quota to get detailed information for
>> all
>>> projects).
>>>
>>> So this is anecdotal. I also get no queue when I submit PR at 11 pm.
>>> Actually whole Airflow committer team had to switch to the "night shift"
>>> because of that. And the most "traffic-heavy" projects - Spark, Pulsar,
>>> Superset, Beam, Airflow - I think some of the top "traffic" projects
>>> experience the same issues and several hours queue when they run during
>> the
>>> EMEA day/US morning. And we all together try to help each other (for
>>> example I helped yesterday the Pulsar team to implement most aggressive
>> way
>>> of cancelling their workflows https://github.com/apache/pulsar/pull/9503
>>> (you can find pretty good explanation why and how it was implemented this
>>> way), also we are working together with the Pulsar team to optimize their
>>> workflow - there is a document
>>>
>>>
>> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
>>> where several peopel are adding their suggestions (including myself based
>>> on Airflow experiences).
>>>
>>> And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
>>> https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you
>> have
>>> not experienced it, especially that you are the only person committing
>>> there. This is hardly representative for other projects that have 100s of
>>> committers and 100s of PRs a day. I am not sure if you are aware of
>>> that, but those are the most valuable projects for the ASF - as those are
>>> the ones that actually build community (Folowing "comunity over code
>>> motto). If you have 3 PRs in 3 months and there aare 200 other projects
>>> using GA, I think yetus is not going to show up in any meaningful
>>> statistics.
>>>
>>> I am not sure if drawing a conclusion from a project that has 3 PRs in 2
>>> months is the best way of drawing conclusions for the overall Apache
>>> organisation. I think drawing a conclusion from experiences of 5 actually
>>> active projects with sometimes even 100 PRs a day is probably better
>>> justified (yep - there are such projects).
>>> So I would probably agree it has little influence on projects that have
>> no
>>> traffic. But enormous influence on projects that actually have traffic.
>> You
>>> have several teams of people scrambling now to somehow manage their CI
>> as
>>> it is unbearable now. Is this serious ? I'd say so.
>>>
>>> When you see Airflow backed up, maybe you should try submitting a
>>> PR to another project yourself to see what happens.
>>>
>>> I am already spending a TON of my private time trying to help others in
>> the
>>> community. I would really appreciate a little help from your side. So
>> maybe
>>> you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET -> 8pm
>>> CET - this is where regularly bottlenecks happen. Please let everyone
>> know
>>> your findings
>>>
>>> J,
>>>
>>>
>>> On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
>>> <aw...@effectivemachines.com.invalid> wrote:
>>>
>>>>
>>>>
>>>>> On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>> I'm not convinced this is true. I have yet to see any of my PRs for
>>>>> "non-big" projects getting queued while Spark, Airflow, others are.
>>> Thus
>>>>> why I think there are only a handful of projects that are getting
>> upset
>>>>> about this but the rest of us are like "meh whatever."
>>>>>
>>>>> Do you have any data on that? Or is it just anecdotal evidence?
>>>>
>>>> Totally anecdotal. Like when I literally ran a Yetus PR during
>>>> the builds meeting as you were complaining about Airflow having an X
>> deep
>>>> queue. My PR ran fine, no pause.
>>>>
>>>>> You can see some analysis and actually even charts here:
>>>>>
>>> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
>>>>
>>>> Yes, and I don't even see Yetus showing up. I wonder how many
>>>> other projects are getting dropped from the dataset....
>>>>
>>>>> Maybe you have a very tiny "PR traffic" and it is mostly in the time
>>> zone
>>>>> that is not affected?
>>>>
>>>> True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
>>>> different though) But if it was one big FIFO queue, our PR jobs would
>>> also
>>>> get queued. They aren't even when I go look at one of the other
>> projects
>>>> that does have queued jobs.
>>>>
>>>> When you see Airflow backed up, maybe you should try
>> submitting a
>>>> PR to another project yourself to see what happens.
>>>>
>>>> All I'm saying is: right now, that document feels like it is
>>>> _greatly_ overstating the problem and now that you point it out,
>> clearly
>>>> dropping data. It is problem, to be sure, but not all GitHub Actions
>>>> projects are suffering. (I wouldn't be surprised if smaller projects
>> are
>>>> actually fast tracked through the build queue in order to avoid a
>> tyranny
>>>> of the majority/resource starvation problem... which would be ironic
>>> given
>>>> how much of an issue that is at the ASF.)
>>>
>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
>
>
> --
> +48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Absolutely agree Matt. Throwing more hardware at "all of the projects" is
definitely not going to help - I was telling that from the beginning - it
is like building free motorways - the more you build, the more traffic
flows and the traffic jams remain. That's why I think reasonable
self-hosted solution that every project owns (including getting the credits
for that) is the only viable solution IMHO - only then you really start
optimising stuff because you own both - the problem and the solution
(and you do not - uncontrollably) impact other projects.
We've just opened-up today the self-hosted solution in Airflow -
announcement from Ash here:
https://lists.apache.org/thread.html/r2e398f86479e4cbfca13c22e4499fb0becdbba20dd9d6d47e1ed30bd%40%3Cdev.airflow.apache.org%3E
and we will be working out any "teething problems", once we are past that,
We are on our way to achieve the goal from the first paragraph - i.e. be
able to control both problem and solution on a per-project basis. And once
we get some learnings - I am sure we will share our solution and findings
more widely with other projects, so that they could apply
similar solutions. This is especially the missing "security piece" which
was a "blocker" so far, but also auto-scaling and tmpfs-optimisation
results (which is a nice side-effect if we can get the 10x improvements in
feedback time eventually (as it seems we can get there).
We love data @Airflow so we will gather some stats that everyone will be
able to analyse and see how much they can gain from - not only the queue
bottleneck removal but also improving the most important (in my opinion)
metrics for the CI - which is feedback time. I personally think in CI there
are are the only two important metrics: reliability and feedback time.
Nothing else (including cost) matters. But If we get all three improved.
that would be something that we will be happy other projects can also
benefit from.
J.
On Tue, Feb 9, 2021 at 3:16 PM Matt Sicker <bo...@gmail.com> wrote:
> To be honest, this sounds exactly like the usual CI problem on every
> platform. As your project scales up, CI becomes a Hard Problem. I don’t
> think throwing hardware at it indefinitely works, though your research here
> is finding most of the useful things.
>
> On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > The report shows only top contenders. And yes - we know it is flawed -
> > because it shows workflows not jobs (if you read the disclaimers - we
> > simply have not enough API calls quota to get detailed information for
> all
> > projects).
> >
> > So this is anecdotal. I also get no queue when I submit PR at 11 pm.
> > Actually whole Airflow committer team had to switch to the "night shift"
> > because of that. And the most "traffic-heavy" projects - Spark, Pulsar,
> > Superset, Beam, Airflow - I think some of the top "traffic" projects
> > experience the same issues and several hours queue when they run during
> the
> > EMEA day/US morning. And we all together try to help each other (for
> > example I helped yesterday the Pulsar team to implement most aggressive
> way
> > of cancelling their workflows https://github.com/apache/pulsar/pull/9503
> > (you can find pretty good explanation why and how it was implemented this
> > way), also we are working together with the Pulsar team to optimize their
> > workflow - there is a document
> >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
> > where several peopel are adding their suggestions (including myself based
> > on Airflow experiences).
> >
> > And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
> > https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you
> have
> > not experienced it, especially that you are the only person committing
> > there. This is hardly representative for other projects that have 100s of
> > committers and 100s of PRs a day. I am not sure if you are aware of
> > that, but those are the most valuable projects for the ASF - as those are
> > the ones that actually build community (Folowing "comunity over code
> > motto). If you have 3 PRs in 3 months and there aare 200 other projects
> > using GA, I think yetus is not going to show up in any meaningful
> > statistics.
> >
> > I am not sure if drawing a conclusion from a project that has 3 PRs in 2
> > months is the best way of drawing conclusions for the overall Apache
> > organisation. I think drawing a conclusion from experiences of 5 actually
> > active projects with sometimes even 100 PRs a day is probably better
> > justified (yep - there are such projects).
> > So I would probably agree it has little influence on projects that have
> no
> > traffic. But enormous influence on projects that actually have traffic.
> You
> > have several teams of people scrambling now to somehow manage their CI
> as
> > it is unbearable now. Is this serious ? I'd say so.
> >
> > When you see Airflow backed up, maybe you should try submitting a
> > PR to another project yourself to see what happens.
> >
> > I am already spending a TON of my private time trying to help others in
> the
> > community. I would really appreciate a little help from your side. So
> maybe
> > you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET -> 8pm
> > CET - this is where regularly bottlenecks happen. Please let everyone
> know
> > your findings
> >
> > J,
> >
> >
> > On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
> > <aw...@effectivemachines.com.invalid> wrote:
> >
> > >
> > >
> > > > On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > >> I'm not convinced this is true. I have yet to see any of my PRs for
> > > > "non-big" projects getting queued while Spark, Airflow, others are.
> > Thus
> > > > why I think there are only a handful of projects that are getting
> upset
> > > > about this but the rest of us are like "meh whatever."
> > > >
> > > > Do you have any data on that? Or is it just anecdotal evidence?
> > >
> > > Totally anecdotal. Like when I literally ran a Yetus PR during
> > > the builds meeting as you were complaining about Airflow having an X
> deep
> > > queue. My PR ran fine, no pause.
> > >
> > > > You can see some analysis and actually even charts here:
> > > >
> > https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
> > >
> > > Yes, and I don't even see Yetus showing up. I wonder how many
> > > other projects are getting dropped from the dataset....
> > >
> > > > Maybe you have a very tiny "PR traffic" and it is mostly in the time
> > zone
> > > > that is not affected?
> > >
> > > True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
> > > different though) But if it was one big FIFO queue, our PR jobs would
> > also
> > > get queued. They aren't even when I go look at one of the other
> projects
> > > that does have queued jobs.
> > >
> > > When you see Airflow backed up, maybe you should try
> submitting a
> > > PR to another project yourself to see what happens.
> > >
> > > All I'm saying is: right now, that document feels like it is
> > > _greatly_ overstating the problem and now that you point it out,
> clearly
> > > dropping data. It is problem, to be sure, but not all GitHub Actions
> > > projects are suffering. (I wouldn't be surprised if smaller projects
> are
> > > actually fast tracked through the build queue in order to avoid a
> tyranny
> > > of the majority/resource starvation problem... which would be ironic
> > given
> > > how much of an issue that is at the ASF.)
> >
> >
> >
> > --
> > +48 660 796 129
> >
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Matt Sicker <bo...@gmail.com>.
To be honest, this sounds exactly like the usual CI problem on every
platform. As your project scales up, CI becomes a Hard Problem. I don’t
think throwing hardware at it indefinitely works, though your research here
is finding most of the useful things.
On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk <ja...@potiuk.com> wrote:
> The report shows only top contenders. And yes - we know it is flawed -
> because it shows workflows not jobs (if you read the disclaimers - we
> simply have not enough API calls quota to get detailed information for all
> projects).
>
> So this is anecdotal. I also get no queue when I submit PR at 11 pm.
> Actually whole Airflow committer team had to switch to the "night shift"
> because of that. And the most "traffic-heavy" projects - Spark, Pulsar,
> Superset, Beam, Airflow - I think some of the top "traffic" projects
> experience the same issues and several hours queue when they run during the
> EMEA day/US morning. And we all together try to help each other (for
> example I helped yesterday the Pulsar team to implement most aggressive way
> of cancelling their workflows https://github.com/apache/pulsar/pull/9503
> (you can find pretty good explanation why and how it was implemented this
> way), also we are working together with the Pulsar team to optimize their
> workflow - there is a document
>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
> where several peopel are adding their suggestions (including myself based
> on Airflow experiences).
>
> And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
> https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you have
> not experienced it, especially that you are the only person committing
> there. This is hardly representative for other projects that have 100s of
> committers and 100s of PRs a day. I am not sure if you are aware of
> that, but those are the most valuable projects for the ASF - as those are
> the ones that actually build community (Folowing "comunity over code
> motto). If you have 3 PRs in 3 months and there aare 200 other projects
> using GA, I think yetus is not going to show up in any meaningful
> statistics.
>
> I am not sure if drawing a conclusion from a project that has 3 PRs in 2
> months is the best way of drawing conclusions for the overall Apache
> organisation. I think drawing a conclusion from experiences of 5 actually
> active projects with sometimes even 100 PRs a day is probably better
> justified (yep - there are such projects).
> So I would probably agree it has little influence on projects that have no
> traffic. But enormous influence on projects that actually have traffic. You
> have several teams of people scrambling now to somehow manage their CI as
> it is unbearable now. Is this serious ? I'd say so.
>
> When you see Airflow backed up, maybe you should try submitting a
> PR to another project yourself to see what happens.
>
> I am already spending a TON of my private time trying to help others in the
> community. I would really appreciate a little help from your side. So maybe
> you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET -> 8pm
> CET - this is where regularly bottlenecks happen. Please let everyone know
> your findings
>
> J,
>
>
> On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
> <aw...@effectivemachines.com.invalid> wrote:
>
> >
> >
> > > On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> I'm not convinced this is true. I have yet to see any of my PRs for
> > > "non-big" projects getting queued while Spark, Airflow, others are.
> Thus
> > > why I think there are only a handful of projects that are getting upset
> > > about this but the rest of us are like "meh whatever."
> > >
> > > Do you have any data on that? Or is it just anecdotal evidence?
> >
> > Totally anecdotal. Like when I literally ran a Yetus PR during
> > the builds meeting as you were complaining about Airflow having an X deep
> > queue. My PR ran fine, no pause.
> >
> > > You can see some analysis and actually even charts here:
> > >
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
> >
> > Yes, and I don't even see Yetus showing up. I wonder how many
> > other projects are getting dropped from the dataset....
> >
> > > Maybe you have a very tiny "PR traffic" and it is mostly in the time
> zone
> > > that is not affected?
> >
> > True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
> > different though) But if it was one big FIFO queue, our PR jobs would
> also
> > get queued. They aren't even when I go look at one of the other projects
> > that does have queued jobs.
> >
> > When you see Airflow backed up, maybe you should try submitting a
> > PR to another project yourself to see what happens.
> >
> > All I'm saying is: right now, that document feels like it is
> > _greatly_ overstating the problem and now that you point it out, clearly
> > dropping data. It is problem, to be sure, but not all GitHub Actions
> > projects are suffering. (I wouldn't be surprised if smaller projects are
> > actually fast tracked through the build queue in order to avoid a tyranny
> > of the majority/resource starvation problem... which would be ironic
> given
> > how much of an issue that is at the ASF.)
>
>
>
> --
> +48 660 796 129
>
Re: GA workflow cancellation
Posted by Jarek Potiuk <ja...@potiuk.com>.
I updated the docs slightly:
* https://github.com/potiuk/cancel-workflow-runs#the-action-target-workflow
-> description about the workflow the action acts on
*
https://github.com/potiuk/cancel-workflow-runs#tackling-the-high-queue-strain-situation
-> I added "Tackling the high queue strain situation" chapter - where I
explain how `allDuplicates` mode helps to fight it.
J
On Tue, Feb 9, 2021 at 7:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > Er... so "source" workflow is the triggering workflow, and "target"
> workflow is the cancelling workflow? Or is it something else? I'm
> afraid all those explanations are a bit confusing to me :-)
>
> Precisely. Unfortunately GA's features for the "special" workflows are as
> powerful as they are complex, and even they did not introduce a good
> terminology. The whole `workflow_run` concept is fantastic from being
> security conscious, but it is also damn difficult to wrap your head around
> it. It took me quite some time to master it and even now I have trouble
> with explaining it, sorry.
> But yes - the terminology I introduced is: PR workflow ("source workflow")
> with READ-ONLY GITHUB_TOKEN is the one that triggers the "workflow_run"
> Cancel workflow ("target workflow") (with WRITE GITHUB_TOKEN).
>
> > I would gladly try to make a PR, but my Javascript competence is
> approximately zero, unfortunately. :-S
>
> I am not at all Javascript dev. If anything I am nowadays mostly a Python
> and Bash developer (often also a Gmail writer ;) ) , but I do have a good
> deal of C/C++/Java/Groovy/Bash and a bit of Javascript under my belt.
> Actually, the action is written in TypeScript not Javascript (Typescript is
> transpiled to javascript) and developing in Typescript if you have a good
> IDE is easy :). This might be easier than you think.
>
> J.
>
> On Tue, Feb 9, 2021 at 7:34 PM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> Le 09/02/2021 à 19:28, Jarek Potiuk a écrit :
>> > Yes. I noticed this misleading line and I will update the docs shortly.
>> It
>> > should be """"Cancels duplicate runs from all running workflow *runs of
>> the
>> > workflow the action acts on*."""
>> >
>> > The action is written in the way that it always acts on a single
>> workflow.
>> >
>> > By default this is indeed the same workflow as the one run by action.
>> But
>> > there are few ways it can be a different workflow:
>> >
>> > 1) In most cases (especially when you want to enable canceling for Pull
>> > Requests from forks), the action should be run "workflow_run" event. And
>> > this even has two workflows "the source one" (for example pull_request)
>> and
>> > "the target one" (this is the workflow that the cancel action should be
>> > part of). If you specify "${{ github.event.workflow_run.id }}" as
>> > sourceRunId - then the action will act on the "Source" workflow instead
>> of
>> > the default "target one".
>>
>> Er... so "source" workflow is the triggering workflow, and "target"
>> workflow is the cancelling workflow? Or is it something else? I'm
>> afraid all those explanations are a bit confusing to me :-)
>>
>> > 2) You can explicitly specify the "workflowFileName" as I did in Pulsar
>> -
>> > then the action will act on the specified workflow.
>> >
>> > It would be possible to add a feature to the action to run on "array of
>> > workflowFileNames" - It is possible, and not very complex, but I decided
>> > not to do it, as it is just a slight improvement over the copy&pasting
>> the
>> > action to workflow file several times (As I did in pulsar).
>>
>> The "array of workflowFileNames" would be quite welcome for us, because
>> right now the amount of copy / pasting is a bit unwiedly, and it's very
>> easy to make an error:
>> https://github.com/apache/arrow/pull/9455/files
>>
>> I would gladly try to make a PR, but my Javascript competence is
>> approximately zero, unfortunately. :-S
>>
>> Best regards
>>
>> Antoine.
>>
>
>
> --
> +48 660 796 129
>
--
+48 660 796 129
Re: GA workflow cancellation
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Er... so "source" workflow is the triggering workflow, and "target"
workflow is the cancelling workflow? Or is it something else? I'm
afraid all those explanations are a bit confusing to me :-)
Precisely. Unfortunately GA's features for the "special" workflows are as
powerful as they are complex, and even they did not introduce a good
terminology. The whole `workflow_run` concept is fantastic from being
security conscious, but it is also damn difficult to wrap your head around
it. It took me quite some time to master it and even now I have trouble
with explaining it, sorry.
But yes - the terminology I introduced is: PR workflow ("source workflow")
with READ-ONLY GITHUB_TOKEN is the one that triggers the "workflow_run"
Cancel workflow ("target workflow") (with WRITE GITHUB_TOKEN).
> I would gladly try to make a PR, but my Javascript competence is
approximately zero, unfortunately. :-S
I am not at all Javascript dev. If anything I am nowadays mostly a Python
and Bash developer (often also a Gmail writer ;) ) , but I do have a good
deal of C/C++/Java/Groovy/Bash and a bit of Javascript under my belt.
Actually, the action is written in TypeScript not Javascript (Typescript is
transpiled to javascript) and developing in Typescript if you have a good
IDE is easy :). This might be easier than you think.
J.
On Tue, Feb 9, 2021 at 7:34 PM Antoine Pitrou <an...@python.org> wrote:
>
> Le 09/02/2021 à 19:28, Jarek Potiuk a écrit :
> > Yes. I noticed this misleading line and I will update the docs shortly.
> It
> > should be """"Cancels duplicate runs from all running workflow *runs of
> the
> > workflow the action acts on*."""
> >
> > The action is written in the way that it always acts on a single
> workflow.
> >
> > By default this is indeed the same workflow as the one run by action. But
> > there are few ways it can be a different workflow:
> >
> > 1) In most cases (especially when you want to enable canceling for Pull
> > Requests from forks), the action should be run "workflow_run" event. And
> > this even has two workflows "the source one" (for example pull_request)
> and
> > "the target one" (this is the workflow that the cancel action should be
> > part of). If you specify "${{ github.event.workflow_run.id }}" as
> > sourceRunId - then the action will act on the "Source" workflow instead
> of
> > the default "target one".
>
> Er... so "source" workflow is the triggering workflow, and "target"
> workflow is the cancelling workflow? Or is it something else? I'm
> afraid all those explanations are a bit confusing to me :-)
>
> > 2) You can explicitly specify the "workflowFileName" as I did in Pulsar -
> > then the action will act on the specified workflow.
> >
> > It would be possible to add a feature to the action to run on "array of
> > workflowFileNames" - It is possible, and not very complex, but I decided
> > not to do it, as it is just a slight improvement over the copy&pasting
> the
> > action to workflow file several times (As I did in pulsar).
>
> The "array of workflowFileNames" would be quite welcome for us, because
> right now the amount of copy / pasting is a bit unwiedly, and it's very
> easy to make an error:
> https://github.com/apache/arrow/pull/9455/files
>
> I would gladly try to make a PR, but my Javascript competence is
> approximately zero, unfortunately. :-S
>
> Best regards
>
> Antoine.
>
--
+48 660 796 129
Re: GA workflow cancellation
Posted by Antoine Pitrou <an...@python.org>.
Le 09/02/2021 à 19:28, Jarek Potiuk a écrit :
> Yes. I noticed this misleading line and I will update the docs shortly. It
> should be """"Cancels duplicate runs from all running workflow *runs of the
> workflow the action acts on*."""
>
> The action is written in the way that it always acts on a single workflow.
>
> By default this is indeed the same workflow as the one run by action. But
> there are few ways it can be a different workflow:
>
> 1) In most cases (especially when you want to enable canceling for Pull
> Requests from forks), the action should be run "workflow_run" event. And
> this even has two workflows "the source one" (for example pull_request) and
> "the target one" (this is the workflow that the cancel action should be
> part of). If you specify "${{ github.event.workflow_run.id }}" as
> sourceRunId - then the action will act on the "Source" workflow instead of
> the default "target one".
Er... so "source" workflow is the triggering workflow, and "target"
workflow is the cancelling workflow? Or is it something else? I'm
afraid all those explanations are a bit confusing to me :-)
> 2) You can explicitly specify the "workflowFileName" as I did in Pulsar -
> then the action will act on the specified workflow.
>
> It would be possible to add a feature to the action to run on "array of
> workflowFileNames" - It is possible, and not very complex, but I decided
> not to do it, as it is just a slight improvement over the copy&pasting the
> action to workflow file several times (As I did in pulsar).
The "array of workflowFileNames" would be quite welcome for us, because
right now the amount of copy / pasting is a bit unwiedly, and it's very
easy to make an error:
https://github.com/apache/arrow/pull/9455/files
I would gladly try to make a PR, but my Javascript competence is
approximately zero, unfortunately. :-S
Best regards
Antoine.
Re: GA workflow cancellation
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yes. I noticed this misleading line and I will update the docs shortly. It
should be """"Cancels duplicate runs from all running workflow *runs of the
workflow the action acts on*."""
The action is written in the way that it always acts on a single workflow.
By default this is indeed the same workflow as the one run by action. But
there are few ways it can be a different workflow:
1) In most cases (especially when you want to enable canceling for Pull
Requests from forks), the action should be run "workflow_run" event. And
this even has two workflows "the source one" (for example pull_request) and
"the target one" (this is the workflow that the cancel action should be
part of). If you specify "${{ github.event.workflow_run.id }}" as
sourceRunId - then the action will act on the "Source" workflow instead of
the default "target one".
2) You can explicitly specify the "workflowFileName" as I did in Pulsar -
then the action will act on the specified workflow.
It would be possible to add a feature to the action to run on "array of
workflowFileNames" - It is possible, and not very complex, but I decided
not to do it, as it is just a slight improvement over the copy&pasting the
action to workflow file several times (As I did in pulsar). The "workflow
per action" however can have the benefit that you can clearly see in the
Gitihub Action output the logs separated per workflow - which makes it
easier to debug and reason about.
However similar results could be achieved by using log grouping
https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#grouping-log-lines
which I did not realize back then. So if someone would like to make PR and
add such feature - I am happy to review and merge it! You could also
potentially query for all the workflows using the API, but this is rather
dangerous IMHO - for example in Pulsar I had to exclude one of the
workflows, because it was only triggered by creating/updating an issue and
my action failed when trying to query it. Probably this could be detected
and fixed, but there are many types of workflows in Github Actions, so
spelling them out explicitly is better Idea IMHO.
Regarding "queuing the cancelling workflow. Indeed, cancel workflows will
be themselves queued - this is why "alllDuplicates" cancel mode was
introduced - I explained exactly how it works and how the "high strain"
situation is handled in this comment:
https://github.com/apache/pulsar/pull/9503#issuecomment-774644408 - I am
moving part of the explanation to the documentation of the action soon.
J.
On Tue, Feb 9, 2021 at 11:43 AM Antoine Pitrou <an...@python.org> wrote:
>
> Hi Jarek,
>
> Le 09/02/2021 à 09:21, Jarek Potiuk a écrit :
> > And we all together try to help each other (for
> > example I helped yesterday the Pulsar team to implement most aggressive
> way
> > of cancelling their workflows https://github.com/apache/pulsar/pull/9503
> > (you can find pretty good explanation why and how it was implemented this
> > way),
>
> Thanks for doing this. I'm trying to experiment with your action and
> there's something a bit confusing in the documentation. It says for the
> "allDuplicates" action:
>
> """Cancels duplicate runs from all running workflows."""
>
> However, it doesn't seem to cancel duplicates from all running
> workflows, only from the triggering workflow. Is that true?
>
> Would there be a way to really cancel duplicates accross all workflows,
> not only the triggering workflow, without listing all the workflow files
> explicitly like you did for Pulsar?
>
>
> Ah, another concern: since the cancelling workflow is itself a workflow,
> it may be subject to the same queueing as other workflows... meaning, if
> the build queue is clogged, it will run too late to have the maximum
> effect?
>
> Thank you
>
> Regards
>
> Antoine.
>
--
+48 660 796 129
Re: GA workflow cancellation
Posted by Antoine Pitrou <an...@python.org>.
Hi Jarek,
Le 09/02/2021 à 09:21, Jarek Potiuk a écrit :
> And we all together try to help each other (for
> example I helped yesterday the Pulsar team to implement most aggressive way
> of cancelling their workflows https://github.com/apache/pulsar/pull/9503
> (you can find pretty good explanation why and how it was implemented this
> way),
Thanks for doing this. I'm trying to experiment with your action and
there's something a bit confusing in the documentation. It says for the
"allDuplicates" action:
"""Cancels duplicate runs from all running workflows."""
However, it doesn't seem to cancel duplicates from all running
workflows, only from the triggering workflow. Is that true?
Would there be a way to really cancel duplicates accross all workflows,
not only the triggering workflow, without listing all the workflow files
explicitly like you did for Pulsar?
Ah, another concern: since the cancelling workflow is itself a workflow,
it may be subject to the same queueing as other workflows... meaning, if
the build queue is clogged, it will run too late to have the maximum effect?
Thank you
Regards
Antoine.
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
The report shows only top contenders. And yes - we know it is flawed -
because it shows workflows not jobs (if you read the disclaimers - we
simply have not enough API calls quota to get detailed information for all
projects).
So this is anecdotal. I also get no queue when I submit PR at 11 pm.
Actually whole Airflow committer team had to switch to the "night shift"
because of that. And the most "traffic-heavy" projects - Spark, Pulsar,
Superset, Beam, Airflow - I think some of the top "traffic" projects
experience the same issues and several hours queue when they run during the
EMEA day/US morning. And we all together try to help each other (for
example I helped yesterday the Pulsar team to implement most aggressive way
of cancelling their workflows https://github.com/apache/pulsar/pull/9503
(you can find pretty good explanation why and how it was implemented this
way), also we are working together with the Pulsar team to optimize their
workflow - there is a document
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit
where several peopel are adding their suggestions (including myself based
on Airflow experiences).
And with yetus' 12 (!) wokflow runs over the last 2 monhts (!)
https://pasteboard.co/JNwGLiR.png - indeed you have a high chance you have
not experienced it, especially that you are the only person committing
there. This is hardly representative for other projects that have 100s of
committers and 100s of PRs a day. I am not sure if you are aware of
that, but those are the most valuable projects for the ASF - as those are
the ones that actually build community (Folowing "comunity over code
motto). If you have 3 PRs in 3 months and there aare 200 other projects
using GA, I think yetus is not going to show up in any meaningful
statistics.
I am not sure if drawing a conclusion from a project that has 3 PRs in 2
months is the best way of drawing conclusions for the overall Apache
organisation. I think drawing a conclusion from experiences of 5 actually
active projects with sometimes even 100 PRs a day is probably better
justified (yep - there are such projects).
So I would probably agree it has little influence on projects that have no
traffic. But enormous influence on projects that actually have traffic. You
have several teams of people scrambling now to somehow manage their CI as
it is unbearable now. Is this serious ? I'd say so.
When you see Airflow backed up, maybe you should try submitting a
PR to another project yourself to see what happens.
I am already spending a TON of my private time trying to help others in the
community. I would really appreciate a little help from your side. So maybe
you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET -> 8pm
CET - this is where regularly bottlenecks happen. Please let everyone know
your findings
J,
On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
>
>
> > On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> I'm not convinced this is true. I have yet to see any of my PRs for
> > "non-big" projects getting queued while Spark, Airflow, others are. Thus
> > why I think there are only a handful of projects that are getting upset
> > about this but the rest of us are like "meh whatever."
> >
> > Do you have any data on that? Or is it just anecdotal evidence?
>
> Totally anecdotal. Like when I literally ran a Yetus PR during
> the builds meeting as you were complaining about Airflow having an X deep
> queue. My PR ran fine, no pause.
>
> > You can see some analysis and actually even charts here:
> > https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
>
> Yes, and I don't even see Yetus showing up. I wonder how many
> other projects are getting dropped from the dataset....
>
> > Maybe you have a very tiny "PR traffic" and it is mostly in the time zone
> > that is not affected?
>
> True, it has very tiny PR traffic right now. (Sep/Oct/Nov was
> different though) But if it was one big FIFO queue, our PR jobs would also
> get queued. They aren't even when I go look at one of the other projects
> that does have queued jobs.
>
> When you see Airflow backed up, maybe you should try submitting a
> PR to another project yourself to see what happens.
>
> All I'm saying is: right now, that document feels like it is
> _greatly_ overstating the problem and now that you point it out, clearly
> dropping data. It is problem, to be sure, but not all GitHub Actions
> projects are suffering. (I wouldn't be surprised if smaller projects are
> actually fast tracked through the build queue in order to avoid a tyranny
> of the majority/resource starvation problem... which would be ironic given
> how much of an issue that is at the ASF.)
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Feb 8, 2021, at 5:00 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I'm not convinced this is true. I have yet to see any of my PRs for
> "non-big" projects getting queued while Spark, Airflow, others are. Thus
> why I think there are only a handful of projects that are getting upset
> about this but the rest of us are like "meh whatever."
>
> Do you have any data on that? Or is it just anecdotal evidence?
Totally anecdotal. Like when I literally ran a Yetus PR during the builds meeting as you were complaining about Airflow having an X deep queue. My PR ran fine, no pause.
> You can see some analysis and actually even charts here:
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
Yes, and I don't even see Yetus showing up. I wonder how many other projects are getting dropped from the dataset....
> Maybe you have a very tiny "PR traffic" and it is mostly in the time zone
> that is not affected?
True, it has very tiny PR traffic right now. (Sep/Oct/Nov was different though) But if it was one big FIFO queue, our PR jobs would also get queued. They aren't even when I go look at one of the other projects that does have queued jobs.
When you see Airflow backed up, maybe you should try submitting a PR to another project yourself to see what happens.
All I'm saying is: right now, that document feels like it is _greatly_ overstating the problem and now that you point it out, clearly dropping data. It is problem, to be sure, but not all GitHub Actions projects are suffering. (I wouldn't be surprised if smaller projects are actually fast tracked through the build queue in order to avoid a tyranny of the majority/resource starvation problem... which would be ironic given how much of an issue that is at the ASF.)
Re: GA again unreasonably slow (again)
Posted by Jarek Potiuk <ja...@potiuk.com>.
> I'm not convinced this is true. I have yet to see any of my PRs for
"non-big" projects getting queued while Spark, Airflow, others are. Thus
why I think there are only a handful of projects that are getting upset
about this but the rest of us are like "meh whatever."
Do you have any data on that? Or is it just anecdotal evidence?
As opposed to anecdotal evidence we build a system to gather statistics on
that - you can see some charts resulting from that showing that we have
regularly started to experience ~300 / ~400 workflows queued during the
EMEA day.
You can see some analysis and actually even charts here:
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
Maybe you have a very tiny "PR traffic" and it is mostly in the time zone
that is not affected?
What project are you talking about Allen? What time zone are you in and how
often your PRs are running in different time-zones ?
Could you please provide some statistics on that ? It seems surprising that
you have different experiences and I would love to get to the bottom of it.
Can you help with understanding why you are not experiencing it? That might
be helpful for others who do.
J
On Tue, Feb 9, 2021 at 1:33 AM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
>
> > On Feb 7, 2021, at 4:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > If you are interested - my document is here. Open for comments - happy to
> > add you as editors if you want (just send me your gmail address in priv).
> > It is rather crude, I had no time to put a bit more effort into it due to
> > some significant changes in my company, but it should be easy to compare
> > the values and see the actual improvements we can get. There are likely a
> > few shortcuts there and some of the numbers are "back-of-the-envelope"
> and
> > we are going to validate them even more when we implement all the
> > optimisations, but the conclusions should be pretty sound.
> >
> >
> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
>
>
>
> "For Apache projects, starting December 2020 we are experiencing a high
> strain of GitHub Actions jobs. All Apache projects are sharing 180 jobs and
> as more projects are using GitHub Actions the job queue becomes a serious
> bottleneck. "
>
> I'm not convinced this is true. I have yet to see any of my PRs
> for "non-big" projects getting queued while Spark, Airflow, others are.
> Thus why I think there are only a handful of projects that are getting
> upset about this but the rest of us are like "meh whatever."
>
>
>
>
--
+48 660 796 129
Re: GA again unreasonably slow (again)
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Feb 7, 2021, at 4:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> If you are interested - my document is here. Open for comments - happy to
> add you as editors if you want (just send me your gmail address in priv).
> It is rather crude, I had no time to put a bit more effort into it due to
> some significant changes in my company, but it should be easy to compare
> the values and see the actual improvements we can get. There are likely a
> few shortcuts there and some of the numbers are "back-of-the-envelope" and
> we are going to validate them even more when we implement all the
> optimisations, but the conclusions should be pretty sound.
>
> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit#
"For Apache projects, starting December 2020 we are experiencing a high strain of GitHub Actions jobs. All Apache projects are sharing 180 jobs and as more projects are using GitHub Actions the job queue becomes a serious bottleneck. "
I'm not convinced this is true. I have yet to see any of my PRs for "non-big" projects getting queued while Spark, Airflow, others are. Thus why I think there are only a handful of projects that are getting upset about this but the rest of us are like "meh whatever."