You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/12/27 15:09:30 UTC
Infinite loop triggered by the Action Policy change
I think we have far bigger problem now.
Seems that the change triggered an infinite loop of some kind and my user
now is disabled from Github Actions. I pushed a change just before the
policy change went live. I guess other people might have similar pro
J.
Hello Jarek,
I work on the Support team at GitHub, and we have some concerns about your
Actions runs on the apache/airflow repository.
We've noticed that there are around 30k failed Actions runs in the last
hour for your potiuk user, and this has been ongoing for a few hours now.
All of those runs are in the apache/airflow repository and can be seen on
this page:
https://github.com/apache/airflow/actions.
It looks like you may have triggered an infinite loop of some kind. As a
result, the repository is consuming a non-trivial amount of resources and
has the potential to negatively affect the GitHub service for other users.
To reduce the number of workflows being run, we've had to temporarily
disable Actions for the potiuk user. We'd ask that you please take a look
at what may have triggered these runs, and once it's resolved we'll be
happy to work with you to turn Actions back on.
Please let us know if you have any questions or concerns.
Thanks,
GitHub Support
Thanks,
Tammy
GitHub Support
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
Ok. The issue is solved for Airflow now. We moved 7 repos in and merged PR
changing the location, enabled workflows back and it seems to be back to
normal - but I think we need to conclude the discussion and make good
learning from the problem.
If anyone has similar problems, GitHub support confirmed that disabling the
workflows solves the "infinite loop" issue.
Not the best Sunday after Xmas I could imagine.
J.
On Sun, Dec 27, 2020 at 5:03 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> FYI. Following Github Advise - I disabled the workflows manually and seems
> that it stopped the infinite loop.
>
> Anyone else having a similar issue should do the same I think.
>
> J
>
>
> On Sun, Dec 27, 2020 at 4:32 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I updated the
>> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
>> think we might have even bigger problem.
>>
>> I think disabling my account did not help.
>>
>> It looks like the infinite loop is still running ... In Airflow we
>> continue having 30K failed jobs/hour...
>>
>> I think you will have to enable it back for a short while or work with
>> the GitHub Support to fix it - I raised the issue to GitHub support, but
>> since it is Sunday after Xmas - I think the easiest way to fix the problem
>> is to disable the policy for a moment so that the infinite loop is stopped
>> (hopefully).
>>
>>
>>
>> On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> I think we have far bigger problem now.
>>>
>>> Seems that the change triggered an infinite loop of some kind and my
>>> user now is disabled from Github Actions. I pushed a change just before the
>>> policy change went live. I guess other people might have similar pro
>>>
>>> J.
>>>
>>>
>>> Hello Jarek,
>>>
>>> I work on the Support team at GitHub, and we have some concerns about
>>> your Actions runs on the apache/airflow repository.
>>>
>>> We've noticed that there are around 30k failed Actions runs in the last
>>> hour for your potiuk user, and this has been ongoing for a few hours
>>> now. All of those runs are in the apache/airflow repository and can be
>>> seen on this page:
>>>
>>> https://github.com/apache/airflow/actions.
>>>
>>> It looks like you may have triggered an infinite loop of some kind. As a
>>> result, the repository is consuming a non-trivial amount of resources and
>>> has the potential to negatively affect the GitHub service for other users.
>>>
>>> To reduce the number of workflows being run, we've had to temporarily
>>> disable Actions for the potiuk user. We'd ask that you please take a
>>> look at what may have triggered these runs, and once it's resolved we'll be
>>> happy to work with you to turn Actions back on.
>>>
>>> Please let us know if you have any questions or concerns.
>>>
>>> Thanks,
>>> GitHub Support
>>>
>>> Thanks,
>>>
>>> Tammy
>>> GitHub Support
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
Ok. The issue is solved for Airflow now. We moved 7 repos in and merged PR
changing the location, enabled workflows back and it seems to be back to
normal - but I think we need to conclude the discussion and make good
learning from the problem.
If anyone has similar problems, GitHub support confirmed that disabling the
workflows solves the "infinite loop" issue.
Not the best Sunday after Xmas I could imagine.
J.
On Sun, Dec 27, 2020 at 5:03 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> FYI. Following Github Advise - I disabled the workflows manually and seems
> that it stopped the infinite loop.
>
> Anyone else having a similar issue should do the same I think.
>
> J
>
>
> On Sun, Dec 27, 2020 at 4:32 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I updated the
>> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
>> think we might have even bigger problem.
>>
>> I think disabling my account did not help.
>>
>> It looks like the infinite loop is still running ... In Airflow we
>> continue having 30K failed jobs/hour...
>>
>> I think you will have to enable it back for a short while or work with
>> the GitHub Support to fix it - I raised the issue to GitHub support, but
>> since it is Sunday after Xmas - I think the easiest way to fix the problem
>> is to disable the policy for a moment so that the infinite loop is stopped
>> (hopefully).
>>
>>
>>
>> On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> I think we have far bigger problem now.
>>>
>>> Seems that the change triggered an infinite loop of some kind and my
>>> user now is disabled from Github Actions. I pushed a change just before the
>>> policy change went live. I guess other people might have similar pro
>>>
>>> J.
>>>
>>>
>>> Hello Jarek,
>>>
>>> I work on the Support team at GitHub, and we have some concerns about
>>> your Actions runs on the apache/airflow repository.
>>>
>>> We've noticed that there are around 30k failed Actions runs in the last
>>> hour for your potiuk user, and this has been ongoing for a few hours
>>> now. All of those runs are in the apache/airflow repository and can be
>>> seen on this page:
>>>
>>> https://github.com/apache/airflow/actions.
>>>
>>> It looks like you may have triggered an infinite loop of some kind. As a
>>> result, the repository is consuming a non-trivial amount of resources and
>>> has the potential to negatively affect the GitHub service for other users.
>>>
>>> To reduce the number of workflows being run, we've had to temporarily
>>> disable Actions for the potiuk user. We'd ask that you please take a
>>> look at what may have triggered these runs, and once it's resolved we'll be
>>> happy to work with you to turn Actions back on.
>>>
>>> Please let us know if you have any questions or concerns.
>>>
>>> Thanks,
>>> GitHub Support
>>>
>>> Thanks,
>>>
>>> Tammy
>>> GitHub Support
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
FYI. Following Github Advise - I disabled the workflows manually and seems
that it stopped the infinite loop.
Anyone else having a similar issue should do the same I think.
J
On Sun, Dec 27, 2020 at 4:32 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I updated the
> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
> think we might have even bigger problem.
>
> I think disabling my account did not help.
>
> It looks like the infinite loop is still running ... In Airflow we
> continue having 30K failed jobs/hour...
>
> I think you will have to enable it back for a short while or work with the
> GitHub Support to fix it - I raised the issue to GitHub support, but since
> it is Sunday after Xmas - I think the easiest way to fix the problem is to
> disable the policy for a moment so that the infinite loop is stopped
> (hopefully).
>
>
>
> On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I think we have far bigger problem now.
>>
>> Seems that the change triggered an infinite loop of some kind and my user
>> now is disabled from Github Actions. I pushed a change just before the
>> policy change went live. I guess other people might have similar pro
>>
>> J.
>>
>>
>> Hello Jarek,
>>
>> I work on the Support team at GitHub, and we have some concerns about
>> your Actions runs on the apache/airflow repository.
>>
>> We've noticed that there are around 30k failed Actions runs in the last
>> hour for your potiuk user, and this has been ongoing for a few hours
>> now. All of those runs are in the apache/airflow repository and can be
>> seen on this page:
>>
>> https://github.com/apache/airflow/actions.
>>
>> It looks like you may have triggered an infinite loop of some kind. As a
>> result, the repository is consuming a non-trivial amount of resources and
>> has the potential to negatively affect the GitHub service for other users.
>>
>> To reduce the number of workflows being run, we've had to temporarily
>> disable Actions for the potiuk user. We'd ask that you please take a
>> look at what may have triggered these runs, and once it's resolved we'll be
>> happy to work with you to turn Actions back on.
>>
>> Please let us know if you have any questions or concerns.
>>
>> Thanks,
>> GitHub Support
>>
>> Thanks,
>>
>> Tammy
>> GitHub Support
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
FYI. Following Github Advise - I disabled the workflows manually and seems
that it stopped the infinite loop.
Anyone else having a similar issue should do the same I think.
J
On Sun, Dec 27, 2020 at 4:32 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I updated the
> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
> think we might have even bigger problem.
>
> I think disabling my account did not help.
>
> It looks like the infinite loop is still running ... In Airflow we
> continue having 30K failed jobs/hour...
>
> I think you will have to enable it back for a short while or work with the
> GitHub Support to fix it - I raised the issue to GitHub support, but since
> it is Sunday after Xmas - I think the easiest way to fix the problem is to
> disable the policy for a moment so that the infinite loop is stopped
> (hopefully).
>
>
>
> On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I think we have far bigger problem now.
>>
>> Seems that the change triggered an infinite loop of some kind and my user
>> now is disabled from Github Actions. I pushed a change just before the
>> policy change went live. I guess other people might have similar pro
>>
>> J.
>>
>>
>> Hello Jarek,
>>
>> I work on the Support team at GitHub, and we have some concerns about
>> your Actions runs on the apache/airflow repository.
>>
>> We've noticed that there are around 30k failed Actions runs in the last
>> hour for your potiuk user, and this has been ongoing for a few hours
>> now. All of those runs are in the apache/airflow repository and can be
>> seen on this page:
>>
>> https://github.com/apache/airflow/actions.
>>
>> It looks like you may have triggered an infinite loop of some kind. As a
>> result, the repository is consuming a non-trivial amount of resources and
>> has the potential to negatively affect the GitHub service for other users.
>>
>> To reduce the number of workflows being run, we've had to temporarily
>> disable Actions for the potiuk user. We'd ask that you please take a
>> look at what may have triggered these runs, and once it's resolved we'll be
>> happy to work with you to turn Actions back on.
>>
>> Please let us know if you have any questions or concerns.
>>
>> Thanks,
>> GitHub Support
>>
>> Thanks,
>>
>> Tammy
>> GitHub Support
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
I updated the
https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
think we might have even bigger problem.
I think disabling my account did not help.
It looks like the infinite loop is still running ... In Airflow we continue
having 30K failed jobs/hour...
I think you will have to enable it back for a short while or work with the
GitHub Support to fix it - I raised the issue to GitHub support, but since
it is Sunday after Xmas - I think the easiest way to fix the problem is to
disable the policy for a moment so that the infinite loop is stopped
(hopefully).
On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I think we have far bigger problem now.
>
> Seems that the change triggered an infinite loop of some kind and my user
> now is disabled from Github Actions. I pushed a change just before the
> policy change went live. I guess other people might have similar pro
>
> J.
>
>
> Hello Jarek,
>
> I work on the Support team at GitHub, and we have some concerns about your
> Actions runs on the apache/airflow repository.
>
> We've noticed that there are around 30k failed Actions runs in the last
> hour for your potiuk user, and this has been ongoing for a few hours now.
> All of those runs are in the apache/airflow repository and can be seen on
> this page:
>
> https://github.com/apache/airflow/actions.
>
> It looks like you may have triggered an infinite loop of some kind. As a
> result, the repository is consuming a non-trivial amount of resources and
> has the potential to negatively affect the GitHub service for other users.
>
> To reduce the number of workflows being run, we've had to temporarily
> disable Actions for the potiuk user. We'd ask that you please take a look
> at what may have triggered these runs, and once it's resolved we'll be
> happy to work with you to turn Actions back on.
>
> Please let us know if you have any questions or concerns.
>
> Thanks,
> GitHub Support
>
> Thanks,
>
> Tammy
> GitHub Support
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Infinite loop triggered by the Action Policy change
Posted by Jarek Potiuk <Ja...@polidea.com>.
I updated the
https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21234 but I
think we might have even bigger problem.
I think disabling my account did not help.
It looks like the infinite loop is still running ... In Airflow we continue
having 30K failed jobs/hour...
I think you will have to enable it back for a short while or work with the
GitHub Support to fix it - I raised the issue to GitHub support, but since
it is Sunday after Xmas - I think the easiest way to fix the problem is to
disable the policy for a moment so that the infinite loop is stopped
(hopefully).
On Sun, Dec 27, 2020 at 4:09 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I think we have far bigger problem now.
>
> Seems that the change triggered an infinite loop of some kind and my user
> now is disabled from Github Actions. I pushed a change just before the
> policy change went live. I guess other people might have similar pro
>
> J.
>
>
> Hello Jarek,
>
> I work on the Support team at GitHub, and we have some concerns about your
> Actions runs on the apache/airflow repository.
>
> We've noticed that there are around 30k failed Actions runs in the last
> hour for your potiuk user, and this has been ongoing for a few hours now.
> All of those runs are in the apache/airflow repository and can be seen on
> this page:
>
> https://github.com/apache/airflow/actions.
>
> It looks like you may have triggered an infinite loop of some kind. As a
> result, the repository is consuming a non-trivial amount of resources and
> has the potential to negatively affect the GitHub service for other users.
>
> To reduce the number of workflows being run, we've had to temporarily
> disable Actions for the potiuk user. We'd ask that you please take a look
> at what may have triggered these runs, and once it's resolved we'll be
> happy to work with you to turn Actions back on.
>
> Please let us know if you have any questions or concerns.
>
> Thanks,
> GitHub Support
>
> Thanks,
>
> Tammy
> GitHub Support
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>