You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Martijn Visser <ma...@ververica.com> on 2022/02/09 09:55:18 UTC
Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Hi everyone,
Please keep in mind that Azure Pipelines currently is dealing with an
incident [1] which causes all CI pipeline runs on Azure to fail. When the
incident has been resolved, it will be required to retrigger your pipeline
to see if the pipeline then passes.
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
[1] https://status.dev.azure.com/_event/287959626
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Johannes Moser <jo...@ververica.com>.
Thanks Martijn and everyone involved.
> On 11.02.2022, at 09:02, Martijn Visser <ma...@ververica.com> wrote:
>
> Hi all,
>
> The issue has been resolved now indeed. The cause was the following:
>
> - Azure had an outage [1]
> - When that outage was resolved, all of Flink's free build agents were
> gone.
> - A temporary switch was made to paid agents, while we reached out once
> more to Azure. However, those paid agents had different/lesser specs,
> causing the e2e_ci runs to fail.
> - It was confirmed that Azure had another outage, causing the free build
> agents to be not available [2]. That outage has now been resolved too.
>
> Everything should work normally now.
>
> Best regards,
>
> Martijn
>
> [1] https://status.dev.azure.com/_event/287959626
> [2] https://status.dev.azure.com/_event/288297230
>
> On Fri, 11 Feb 2022 at 06:56, Biao Geng <bi...@gmail.com> wrote:
>
>> Just notice that there are some recent pipelines that have finished
>> successfully[1]. It seems that this issue is mitigated.
>>
>> [1]
>>
>> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31194&view=results
>>
>> Martijn Visser <ma...@ververica.com> 于2022年2月10日周四 15:48写道：
>>
>>> It looks like the issues haven't been resolved yet unfortunately.
>>>
>>> On Wed, 9 Feb 2022 at 13:15, Robert Metzger <me...@gmail.com> wrote:
>>>
>>>> I filed a support request with Microsoft:
>>>>
>>>>
>>>
>> https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
>>>>
>>>> On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <ma...@ververica.com>
>>>> wrote:
>>>>
>>>>> Unfortunately it looks like there are still failures. Will keep you
>>>> posted
>>>>>
>>>>> On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com>
>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> The issue should now be resolved.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Wed, 9 Feb 2022 at 10:55, Martijn Visser <martijn@ververica.com
>>>
>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Please keep in mind that Azure Pipelines currently is dealing with
>>> an
>>>>>>> incident [1] which causes all CI pipeline runs on Azure to fail.
>>> When
>>>>> the
>>>>>>> incident has been resolved, it will be required to retrigger your
>>>>> pipeline
>>>>>>> to see if the pipeline then passes.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Martijn Visser
>>>>>>> https://twitter.com/MartijnVisser82
>>>>>>>
>>>>>>> [1] https://status.dev.azure.com/_event/287959626
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Konstantin Knauf <kn...@apache.org>.
Thanks for following up on this, Martijn.
On Fri, Feb 11, 2022 at 9:03 AM Martijn Visser <ma...@ververica.com>
wrote:
> Hi all,
>
> The issue has been resolved now indeed. The cause was the following:
>
> - Azure had an outage [1]
> - When that outage was resolved, all of Flink's free build agents were
> gone.
> - A temporary switch was made to paid agents, while we reached out once
> more to Azure. However, those paid agents had different/lesser specs,
> causing the e2e_ci runs to fail.
> - It was confirmed that Azure had another outage, causing the free build
> agents to be not available [2]. That outage has now been resolved too.
>
> Everything should work normally now.
>
> Best regards,
>
> Martijn
>
> [1] https://status.dev.azure.com/_event/287959626
> [2] https://status.dev.azure.com/_event/288297230
>
> On Fri, 11 Feb 2022 at 06:56, Biao Geng <bi...@gmail.com> wrote:
>
> > Just notice that there are some recent pipelines that have finished
> > successfully[1]. It seems that this issue is mitigated.
> >
> > [1]
> >
> >
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31194&view=results
> >
> > Martijn Visser <ma...@ververica.com> 于2022年2月10日周四 15:48写道：
> >
> > > It looks like the issues haven't been resolved yet unfortunately.
> > >
> > > On Wed, 9 Feb 2022 at 13:15, Robert Metzger <me...@gmail.com>
> wrote:
> > >
> > > > I filed a support request with Microsoft:
> > > >
> > > >
> > >
> >
> https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
> > > >
> > > > On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <martijn@ververica.com
> >
> > > > wrote:
> > > >
> > > > > Unfortunately it looks like there are still failures. Will keep you
> > > > posted
> > > > >
> > > > > On Wed, 9 Feb 2022 at 11:51, Martijn Visser <martijn@ververica.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > The issue should now be resolved.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Wed, 9 Feb 2022 at 10:55, Martijn Visser <
> martijn@ververica.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> Please keep in mind that Azure Pipelines currently is dealing
> with
> > > an
> > > > > >> incident [1] which causes all CI pipeline runs on Azure to fail.
> > > When
> > > > > the
> > > > > >> incident has been resolved, it will be required to retrigger
> your
> > > > > pipeline
> > > > > >> to see if the pipeline then passes.
> > > > > >>
> > > > > >> Best regards,
> > > > > >>
> > > > > >> Martijn Visser
> > > > > >> https://twitter.com/MartijnVisser82
> > > > > >>
> > > > > >> [1] https://status.dev.azure.com/_event/287959626
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Konstantin Knauf
https://twitter.com/snntrable
https://github.com/knaufk
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Martijn Visser <ma...@ververica.com>.
Hi all,
The issue has been resolved now indeed. The cause was the following:
- Azure had an outage [1]
- When that outage was resolved, all of Flink's free build agents were
gone.
- A temporary switch was made to paid agents, while we reached out once
more to Azure. However, those paid agents had different/lesser specs,
causing the e2e_ci runs to fail.
- It was confirmed that Azure had another outage, causing the free build
agents to be not available [2]. That outage has now been resolved too.
Everything should work normally now.
Best regards,
Martijn
[1] https://status.dev.azure.com/_event/287959626
[2] https://status.dev.azure.com/_event/288297230
On Fri, 11 Feb 2022 at 06:56, Biao Geng <bi...@gmail.com> wrote:
> Just notice that there are some recent pipelines that have finished
> successfully[1]. It seems that this issue is mitigated.
>
> [1]
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31194&view=results
>
> Martijn Visser <ma...@ververica.com> 于2022年2月10日周四 15:48写道：
>
> > It looks like the issues haven't been resolved yet unfortunately.
> >
> > On Wed, 9 Feb 2022 at 13:15, Robert Metzger <me...@gmail.com> wrote:
> >
> > > I filed a support request with Microsoft:
> > >
> > >
> >
> https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
> > >
> > > On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <ma...@ververica.com>
> > > wrote:
> > >
> > > > Unfortunately it looks like there are still failures. Will keep you
> > > posted
> > > >
> > > > On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com>
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > The issue should now be resolved.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Wed, 9 Feb 2022 at 10:55, Martijn Visser <martijn@ververica.com
> >
> > > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> Please keep in mind that Azure Pipelines currently is dealing with
> > an
> > > > >> incident [1] which causes all CI pipeline runs on Azure to fail.
> > When
> > > > the
> > > > >> incident has been resolved, it will be required to retrigger your
> > > > pipeline
> > > > >> to see if the pipeline then passes.
> > > > >>
> > > > >> Best regards,
> > > > >>
> > > > >> Martijn Visser
> > > > >> https://twitter.com/MartijnVisser82
> > > > >>
> > > > >> [1] https://status.dev.azure.com/_event/287959626
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Biao Geng <bi...@gmail.com>.
Just notice that there are some recent pipelines that have finished
successfully[1]. It seems that this issue is mitigated.
[1]
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31194&view=results
Martijn Visser <ma...@ververica.com> 于2022年2月10日周四 15:48写道：
> It looks like the issues haven't been resolved yet unfortunately.
>
> On Wed, 9 Feb 2022 at 13:15, Robert Metzger <me...@gmail.com> wrote:
>
> > I filed a support request with Microsoft:
> >
> >
> https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
> >
> > On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <ma...@ververica.com>
> > wrote:
> >
> > > Unfortunately it looks like there are still failures. Will keep you
> > posted
> > >
> > > On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > The issue should now be resolved.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Wed, 9 Feb 2022 at 10:55, Martijn Visser <ma...@ververica.com>
> > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> Please keep in mind that Azure Pipelines currently is dealing with
> an
> > > >> incident [1] which causes all CI pipeline runs on Azure to fail.
> When
> > > the
> > > >> incident has been resolved, it will be required to retrigger your
> > > pipeline
> > > >> to see if the pipeline then passes.
> > > >>
> > > >> Best regards,
> > > >>
> > > >> Martijn Visser
> > > >> https://twitter.com/MartijnVisser82
> > > >>
> > > >> [1] https://status.dev.azure.com/_event/287959626
> > > >>
> > > >
> > >
> >
>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Martijn Visser <ma...@ververica.com>.
It looks like the issues haven't been resolved yet unfortunately.
On Wed, 9 Feb 2022 at 13:15, Robert Metzger <me...@gmail.com> wrote:
> I filed a support request with Microsoft:
>
> https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
>
> On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <ma...@ververica.com>
> wrote:
>
> > Unfortunately it looks like there are still failures. Will keep you
> posted
> >
> > On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > The issue should now be resolved.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, 9 Feb 2022 at 10:55, Martijn Visser <ma...@ververica.com>
> > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Please keep in mind that Azure Pipelines currently is dealing with an
> > >> incident [1] which causes all CI pipeline runs on Azure to fail. When
> > the
> > >> incident has been resolved, it will be required to retrigger your
> > pipeline
> > >> to see if the pipeline then passes.
> > >>
> > >> Best regards,
> > >>
> > >> Martijn Visser
> > >> https://twitter.com/MartijnVisser82
> > >>
> > >> [1] https://status.dev.azure.com/_event/287959626
> > >>
> > >
> >
>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Robert Metzger <me...@gmail.com>.
I filed a support request with Microsoft:
https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email&space=21&sort=newest
On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser <ma...@ververica.com> wrote:
> Unfortunately it looks like there are still failures. Will keep you posted
>
> On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com> wrote:
>
> > Hi everyone,
> >
> > The issue should now be resolved.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, 9 Feb 2022 at 10:55, Martijn Visser <ma...@ververica.com>
> wrote:
> >
> >> Hi everyone,
> >>
> >> Please keep in mind that Azure Pipelines currently is dealing with an
> >> incident [1] which causes all CI pipeline runs on Azure to fail. When
> the
> >> incident has been resolved, it will be required to retrigger your
> pipeline
> >> to see if the pipeline then passes.
> >>
> >> Best regards,
> >>
> >> Martijn Visser
> >> https://twitter.com/MartijnVisser82
> >>
> >> [1] https://status.dev.azure.com/_event/287959626
> >>
> >
>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Martijn Visser <ma...@ververica.com>.
Unfortunately it looks like there are still failures. Will keep you posted
On Wed, 9 Feb 2022 at 11:51, Martijn Visser <ma...@ververica.com> wrote:
> Hi everyone,
>
> The issue should now be resolved.
>
> Best regards,
>
> Martijn
>
> On Wed, 9 Feb 2022 at 10:55, Martijn Visser <ma...@ververica.com> wrote:
>
>> Hi everyone,
>>
>> Please keep in mind that Azure Pipelines currently is dealing with an
>> incident [1] which causes all CI pipeline runs on Azure to fail. When the
>> incident has been resolved, it will be required to retrigger your pipeline
>> to see if the pipeline then passes.
>>
>> Best regards,
>>
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>>
>> [1] https://status.dev.azure.com/_event/287959626
>>
>
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Posted by Martijn Visser <ma...@ververica.com>.
Hi everyone,
The issue should now be resolved.
Best regards,
Martijn
On Wed, 9 Feb 2022 at 10:55, Martijn Visser <ma...@ververica.com> wrote:
> Hi everyone,
>
> Please keep in mind that Azure Pipelines currently is dealing with an
> incident [1] which causes all CI pipeline runs on Azure to fail. When the
> incident has been resolved, it will be required to retrigger your pipeline
> to see if the pipeline then passes.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://status.dev.azure.com/_event/287959626
>