You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Chris Lambertus <cm...@apache.org> on 2021/02/17 23:31:31 UTC
ci-builds.apache.org offline - network issue
Hi, ci-builds.apache.org <http://ci-builds.apache.org/> is currently offline due to some kind of network issue. I’ve opened a support ticket with our hosting provider.
Re: ci-builds.apache.org offline - network issue
Posted by Robbie Gemmell <ro...@gmail.com>.
I was about to report this. I see around 30 jobs are missing in our
case, with only 5 actually remaining.
On Thu, 18 Feb 2021 at 05:05, Gavin McDonald <gm...@apache.org> wrote:
>
> Hi All,
>
> Whilst the service is back, there are quite a few jobs 'missing' since the
> machine went offline.
>
> We are investigating.
>
>
> On Thu, Feb 18, 2021 at 12:31 AM Chris Lambertus <cm...@apache.org> wrote:
>
> >
> > Hi, ci-builds.apache.org <http://ci-builds.apache.org/> is currently
> > offline due to some kind of network issue. I’ve opened a support ticket
> > with our hosting provider.
> >
> >
> >
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
Re: ci-builds.apache.org offline - network issue
Posted by Lukasz Lenart <lu...@apache.org>.
Bravo, good work!
pt., 19 lut 2021 o 13:23 Gavin McDonald <gm...@apache.org> napisał(a):
>
> Sorry, somehow missed the S* projects, Sling is there now.
>
> On Fri, Feb 19, 2021 at 12:59 PM Bertrand Delacretaz <bd...@apache.org>
> wrote:
>
> > Hi,
> >
> > On Fri, Feb 19, 2021 at 12:44 PM Gavin McDonald <gm...@apache.org>
> > wrote:
> > > All jobs should be restored; and I have re-enabled the building of
> > jobs...
> >
> > https://ci-builds.apache.org/job/Sling/ stills returns 404 for me. Our
> > jobs are all generated, maybe we need to rerun something to recreate
> > them?
> >
> > If you think that's the case, don't strain to find out exactly how, I
> > can check that with Sling project members, I don't remember off the
> > top of my head.
> >
> > -Bertrand
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
Re: ci-builds.apache.org offline - network issue
Posted by Gavin McDonald <gm...@apache.org>.
Sorry, somehow missed the S* projects, Sling is there now.
On Fri, Feb 19, 2021 at 12:59 PM Bertrand Delacretaz <bd...@apache.org>
wrote:
> Hi,
>
> On Fri, Feb 19, 2021 at 12:44 PM Gavin McDonald <gm...@apache.org>
> wrote:
> > All jobs should be restored; and I have re-enabled the building of
> jobs...
>
> https://ci-builds.apache.org/job/Sling/ stills returns 404 for me. Our
> jobs are all generated, maybe we need to rerun something to recreate
> them?
>
> If you think that's the case, don't strain to find out exactly how, I
> can check that with Sling project members, I don't remember off the
> top of my head.
>
> -Bertrand
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: ci-builds.apache.org offline - network issue
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Fri, Feb 19, 2021 at 12:44 PM Gavin McDonald <gm...@apache.org> wrote:
> All jobs should be restored; and I have re-enabled the building of jobs...
https://ci-builds.apache.org/job/Sling/ stills returns 404 for me. Our
jobs are all generated, maybe we need to rerun something to recreate
them?
If you think that's the case, don't strain to find out exactly how, I
can check that with Sling project members, I don't remember off the
top of my head.
-Bertrand
Re: ci-builds.apache.org offline - network issue
Posted by Gavin McDonald <gm...@apache.org>.
Hi All,
All jobs should be restored; and I have re-enabled the building of jobs.
Please check your projects jobs and let us know if you see any issues.
On Fri, Feb 19, 2021 at 1:42 AM Chris Lambertus <cm...@apache.org> wrote:
>
> Hi folks,
>
> As we continue our investigation, here is an update:
>
> - A number of job config.xml and “folder configuration” xml files appear
> to have vanished in a yet-to-be-determined time window. We believe this
> would have happened between the last Jenkins restart and 17 Feb. We are
> still investigating the exact time window, but our priority has been on
> restoring backups.
>
> - Due to the fact that Jenkins caches the config.xml files while running,
> the problem was not noticed until the service unexpectedly
> crashed/restarted on 17 Feb.
>
> - Unfortunately, due to the extremely large jobs/ directory on ci-builds
> (1.5TB), we are not able to retain many historical backups. At this point,
> we believe we will be able to recover the vast majority of job configs,
> although some may revert to a configuration from early January. We continue
> to work on data restoration, and are taking great care to not overwrite any
> new configurations with old backup data.
>
> Infra is working to evaluate and implement a more robust backup strategy
> for job configurations based on the lessons learned in this event.
>
> Due to the extremely large size of the backup sets, this process will be
> ongoing for at least another 24 hours. We’ll provide another update as soon
> as possible within that timeframe.
>
> -Chris
> ASF Infra
>
>
>
>
> > On Feb 18, 2021, at 11:09 AM, Chris Lambertus <cm...@apache.org> wrote:
> >
> > On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified
> that the ci-builds Jenkins service was offline. We contacted our service
> provider, and they restarted the system. After this restart, Infra started
> receiving notifications of missing job configurations.
> >
> > Upon beginning the investigation, the system again went offline, and
> required another forced restart.
> >
> > Infra's initial investigation shows a number of missing config.xml files
> which we are in the process of restoring from backup.
> >
> > Additionally, Infra is evaluating the existing hardware for issues, and
> beginning a migration process to new hardware.
> >
> > Please bear with us as we restore the config.xml files, and investigate
> the root cause of their disappearance. As part of this recovery, we are
> exploring additional avenues to improve the backup methodologies for job
> configurations.
> >
> > We will provide another update once we are further along in the
> restoration process, likely within 24 hours.
> >
> > -Chris
> > ASF Infra
> >
> >
> >
> >> On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gm...@apache.org>
> wrote:
> >>
> >> Hi All,
> >>
> >> No need for any more messages, all of it is known.
> >>
> >> On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <po...@apache.org>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
> >>>> Whilst the service is back, there are quite a few jobs 'missing' since
> >>> the
> >>>> machine went offline.
> >>>
> >>> Creadur is missing completely:
> >>> https://ci-builds.apache.org/job/Creadur/
> >>>
> >>> :(
> >>>
> >>> Thanks,
> >>> Phil
> >>>
> >>
> >>
> >> --
> >>
> >> *Gavin McDonald*
> >> Systems Administrator
> >> ASF Infrastructure Team
> >
>
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Fwd: ci-builds.apache.org offline - network issue
Posted by Zoran Regvart <zo...@regvart.com>.
FYI
---------- Forwarded message ---------
From: Chris Lambertus <cm...@apache.org>
Date: Fri, Feb 19, 2021 at 1:42 AM
Subject: Re: ci-builds.apache.org offline - network issue
To: builds <bu...@apache.org>
Cc: Gavin McDonald <gm...@apache.org>
Hi folks,
As we continue our investigation, here is an update:
- A number of job config.xml and “folder configuration” xml files
appear to have vanished in a yet-to-be-determined time window. We
believe this would have happened between the last Jenkins restart and
17 Feb. We are still investigating the exact time window, but our
priority has been on restoring backups.
- Due to the fact that Jenkins caches the config.xml files while
running, the problem was not noticed until the service unexpectedly
crashed/restarted on 17 Feb.
- Unfortunately, due to the extremely large jobs/ directory on
ci-builds (1.5TB), we are not able to retain many historical backups.
At this point, we believe we will be able to recover the vast majority
of job configs, although some may revert to a configuration from early
January. We continue to work on data restoration, and are taking great
care to not overwrite any new configurations with old backup data.
Infra is working to evaluate and implement a more robust backup
strategy for job configurations based on the lessons learned in this
event.
Due to the extremely large size of the backup sets, this process will
be ongoing for at least another 24 hours. We’ll provide another update
as soon as possible within that timeframe.
-Chris
ASF Infra
> On Feb 18, 2021, at 11:09 AM, Chris Lambertus <cm...@apache.org> wrote:
>
> On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified that the ci-builds Jenkins service was offline. We contacted our service provider, and they restarted the system. After this restart, Infra started receiving notifications of missing job configurations.
>
> Upon beginning the investigation, the system again went offline, and required another forced restart.
>
> Infra's initial investigation shows a number of missing config.xml files which we are in the process of restoring from backup.
>
> Additionally, Infra is evaluating the existing hardware for issues, and beginning a migration process to new hardware.
>
> Please bear with us as we restore the config.xml files, and investigate the root cause of their disappearance. As part of this recovery, we are exploring additional avenues to improve the backup methodologies for job configurations.
>
> We will provide another update once we are further along in the restoration process, likely within 24 hours.
>
> -Chris
> ASF Infra
>
>
>
>> On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gm...@apache.org> wrote:
>>
>> Hi All,
>>
>> No need for any more messages, all of it is known.
>>
>> On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <po...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
>>>> Whilst the service is back, there are quite a few jobs 'missing' since
>>> the
>>>> machine went offline.
>>>
>>> Creadur is missing completely:
>>> https://ci-builds.apache.org/job/Creadur/
>>>
>>> :(
>>>
>>> Thanks,
>>> Phil
>>>
>>
>>
>> --
>>
>> *Gavin McDonald*
>> Systems Administrator
>> ASF Infrastructure Team
>
--
Zoran Regvart
Re: ci-builds.apache.org offline - network issue
Posted by Chris Lambertus <cm...@apache.org>.
Hi folks,
As we continue our investigation, here is an update:
- A number of job config.xml and “folder configuration” xml files appear to have vanished in a yet-to-be-determined time window. We believe this would have happened between the last Jenkins restart and 17 Feb. We are still investigating the exact time window, but our priority has been on restoring backups.
- Due to the fact that Jenkins caches the config.xml files while running, the problem was not noticed until the service unexpectedly crashed/restarted on 17 Feb.
- Unfortunately, due to the extremely large jobs/ directory on ci-builds (1.5TB), we are not able to retain many historical backups. At this point, we believe we will be able to recover the vast majority of job configs, although some may revert to a configuration from early January. We continue to work on data restoration, and are taking great care to not overwrite any new configurations with old backup data.
Infra is working to evaluate and implement a more robust backup strategy for job configurations based on the lessons learned in this event.
Due to the extremely large size of the backup sets, this process will be ongoing for at least another 24 hours. We’ll provide another update as soon as possible within that timeframe.
-Chris
ASF Infra
> On Feb 18, 2021, at 11:09 AM, Chris Lambertus <cm...@apache.org> wrote:
>
> On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified that the ci-builds Jenkins service was offline. We contacted our service provider, and they restarted the system. After this restart, Infra started receiving notifications of missing job configurations.
>
> Upon beginning the investigation, the system again went offline, and required another forced restart.
>
> Infra's initial investigation shows a number of missing config.xml files which we are in the process of restoring from backup.
>
> Additionally, Infra is evaluating the existing hardware for issues, and beginning a migration process to new hardware.
>
> Please bear with us as we restore the config.xml files, and investigate the root cause of their disappearance. As part of this recovery, we are exploring additional avenues to improve the backup methodologies for job configurations.
>
> We will provide another update once we are further along in the restoration process, likely within 24 hours.
>
> -Chris
> ASF Infra
>
>
>
>> On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gm...@apache.org> wrote:
>>
>> Hi All,
>>
>> No need for any more messages, all of it is known.
>>
>> On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <po...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
>>>> Whilst the service is back, there are quite a few jobs 'missing' since
>>> the
>>>> machine went offline.
>>>
>>> Creadur is missing completely:
>>> https://ci-builds.apache.org/job/Creadur/
>>>
>>> :(
>>>
>>> Thanks,
>>> Phil
>>>
>>
>>
>> --
>>
>> *Gavin McDonald*
>> Systems Administrator
>> ASF Infrastructure Team
>
Re: ci-builds.apache.org offline - network issue
Posted by Chris Lambertus <cm...@apache.org>.
On Wednesday 18 Feb approximately 1800 hours UTC, Infra was notified that the ci-builds Jenkins service was offline. We contacted our service provider, and they restarted the system. After this restart, Infra started receiving notifications of missing job configurations.
Upon beginning the investigation, the system again went offline, and required another forced restart.
Infra's initial investigation shows a number of missing config.xml files which we are in the process of restoring from backup.
Additionally, Infra is evaluating the existing hardware for issues, and beginning a migration process to new hardware.
Please bear with us as we restore the config.xml files, and investigate the root cause of their disappearance. As part of this recovery, we are exploring additional avenues to improve the backup methodologies for job configurations.
We will provide another update once we are further along in the restoration process, likely within 24 hours.
-Chris
ASF Infra
> On Feb 18, 2021, at 1:20 AM, Gavin McDonald <gm...@apache.org> wrote:
>
> Hi All,
>
> No need for any more messages, all of it is known.
>
> On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <po...@apache.org> wrote:
>
>> Hi,
>>
>> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
>>> Whilst the service is back, there are quite a few jobs 'missing' since
>> the
>>> machine went offline.
>>
>> Creadur is missing completely:
>> https://ci-builds.apache.org/job/Creadur/
>>
>> :(
>>
>> Thanks,
>> Phil
>>
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
Re: ci-builds.apache.org offline - network issue
Posted by Gavin McDonald <gm...@apache.org>.
Hi All,
No need for any more messages, all of it is known.
On Thu, Feb 18, 2021 at 10:15 AM P. Ottlinger <po...@apache.org> wrote:
> Hi,
>
> Am 18.02.21 um 06:05 schrieb Gavin McDonald:
> > Whilst the service is back, there are quite a few jobs 'missing' since
> the
> > machine went offline.
>
> Creadur is missing completely:
> https://ci-builds.apache.org/job/Creadur/
>
> :(
>
> Thanks,
> Phil
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: ci-builds.apache.org offline - network issue
Posted by "P. Ottlinger" <po...@apache.org>.
Hi,
Am 18.02.21 um 06:05 schrieb Gavin McDonald:
> Whilst the service is back, there are quite a few jobs 'missing' since the
> machine went offline.
Creadur is missing completely:
https://ci-builds.apache.org/job/Creadur/
:(
Thanks,
Phil
Re: ci-builds.apache.org offline - network issue
Posted by Gavin McDonald <gm...@apache.org>.
Hi All,
Whilst the service is back, there are quite a few jobs 'missing' since the
machine went offline.
We are investigating.
On Thu, Feb 18, 2021 at 12:31 AM Chris Lambertus <cm...@apache.org> wrote:
>
> Hi, ci-builds.apache.org <http://ci-builds.apache.org/> is currently
> offline due to some kind of network issue. I’ve opened a support ticket
> with our hosting provider.
>
>
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team