You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Jarek Jarcec Cecho <ja...@apache.org> on 2015/08/21 18:05:45 UTC
What is the process of reporting dead slaves
Looking at Jenkins queue - there is 132 queued jobs and only 4 working executors :) All the others are marked as “DEAD”.
I’m wondering what is the process of reporting that? Should I create BUILDS or INFRA JIRA, or just send email to this group or something completely else?
Jarcec
Re: What is the process of reporting dead slaves
Posted by Andrew Bayer <an...@gmail.com>.
We ran out of disk space on builds.a.o - I'm compressing the 487gb (!!!) of
logs and will work with the rest of Infra to make sure that's done
automatically.
A.
On Fri, Aug 21, 2015 at 12:18 PM, Andrew Bayer <an...@gmail.com>
wrote:
> I'll take a look in 30 minutes or so.
> On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>
>> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
>> executors :) All the others are marked as “DEAD”.
>>
>> I’m wondering what is the process of reporting that? Should I create
>> BUILDS or INFRA JIRA, or just send email to this group or something
>> completely else?
>>
>> Jarcec
>
>
Re: What is the process of reporting dead slaves
Posted by Andrew Bayer <an...@gmail.com>.
Everything's lively again now - I'll make sure we've got disk space
monitoring in place again.
A.
On Fri, Aug 21, 2015 at 12:33 PM, Andrew Bayer <an...@gmail.com>
wrote:
> It's not the slaves, it's the master itself.
>
> A.
>
> On Fri, Aug 21, 2015 at 12:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
> wrote:
>
>> Thanks Andrew!
>>
>> Just to give you heads up - it just seems that the build slaves went out
>> of disk space as I see this exception:
>>
>> java.io.IOException: No space left on device
>>
>> (probably some rogue job or something)
>>
>> Jarcec
>>
>> > On Aug 21, 2015, at 9:18 AM, Andrew Bayer <an...@gmail.com>
>> wrote:
>> >
>> > I'll take a look in 30 minutes or so.
>> > On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>> >
>> >> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
>> >> executors :) All the others are marked as “DEAD”.
>> >>
>> >> I’m wondering what is the process of reporting that? Should I create
>> >> BUILDS or INFRA JIRA, or just send email to this group or something
>> >> completely else?
>> >>
>> >> Jarcec
>>
>>
>
Re: What is the process of reporting dead slaves
Posted by Andrew Bayer <an...@gmail.com>.
FWIW, https://issues.apache.org/jira/browse/INFRA-10171 will help us catch
this earlier, and https://issues.apache.org/jira/browse/INFRA-10172 will
help keep it from happening in the first place. I'm also looking to see
what jobs are using the most disk space to try to trim their usage down.
On Fri, Aug 21, 2015 at 12:56 PM, Jarek Jarcec Cecho <ja...@apache.org>
wrote:
> Got it, it looks much better now - thank you!
>
> Jarcec
>
> > On Aug 21, 2015, at 9:33 AM, Andrew Bayer <an...@gmail.com>
> wrote:
> >
> > It's not the slaves, it's the master itself.
> >
> > A.
> >
> > On Fri, Aug 21, 2015 at 12:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
> > wrote:
> >
> >> Thanks Andrew!
> >>
> >> Just to give you heads up - it just seems that the build slaves went out
> >> of disk space as I see this exception:
> >>
> >> java.io.IOException: No space left on device
> >>
> >> (probably some rogue job or something)
> >>
> >> Jarcec
> >>
> >>> On Aug 21, 2015, at 9:18 AM, Andrew Bayer <an...@gmail.com>
> >> wrote:
> >>>
> >>> I'll take a look in 30 minutes or so.
> >>> On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
> >>>
> >>>> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
> >>>> executors :) All the others are marked as “DEAD”.
> >>>>
> >>>> I’m wondering what is the process of reporting that? Should I create
> >>>> BUILDS or INFRA JIRA, or just send email to this group or something
> >>>> completely else?
> >>>>
> >>>> Jarcec
> >>
> >>
>
>
Re: What is the process of reporting dead slaves
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Got it, it looks much better now - thank you!
Jarcec
> On Aug 21, 2015, at 9:33 AM, Andrew Bayer <an...@gmail.com> wrote:
>
> It's not the slaves, it's the master itself.
>
> A.
>
> On Fri, Aug 21, 2015 at 12:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
> wrote:
>
>> Thanks Andrew!
>>
>> Just to give you heads up - it just seems that the build slaves went out
>> of disk space as I see this exception:
>>
>> java.io.IOException: No space left on device
>>
>> (probably some rogue job or something)
>>
>> Jarcec
>>
>>> On Aug 21, 2015, at 9:18 AM, Andrew Bayer <an...@gmail.com>
>> wrote:
>>>
>>> I'll take a look in 30 minutes or so.
>>> On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>>>
>>>> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
>>>> executors :) All the others are marked as “DEAD”.
>>>>
>>>> I’m wondering what is the process of reporting that? Should I create
>>>> BUILDS or INFRA JIRA, or just send email to this group or something
>>>> completely else?
>>>>
>>>> Jarcec
>>
>>
Re: What is the process of reporting dead slaves
Posted by Andrew Bayer <an...@gmail.com>.
It's not the slaves, it's the master itself.
A.
On Fri, Aug 21, 2015 at 12:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
wrote:
> Thanks Andrew!
>
> Just to give you heads up - it just seems that the build slaves went out
> of disk space as I see this exception:
>
> java.io.IOException: No space left on device
>
> (probably some rogue job or something)
>
> Jarcec
>
> > On Aug 21, 2015, at 9:18 AM, Andrew Bayer <an...@gmail.com>
> wrote:
> >
> > I'll take a look in 30 minutes or so.
> > On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
> >
> >> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
> >> executors :) All the others are marked as “DEAD”.
> >>
> >> I’m wondering what is the process of reporting that? Should I create
> >> BUILDS or INFRA JIRA, or just send email to this group or something
> >> completely else?
> >>
> >> Jarcec
>
>
Re: What is the process of reporting dead slaves
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Thanks Andrew!
Just to give you heads up - it just seems that the build slaves went out of disk space as I see this exception:
java.io.IOException: No space left on device
(probably some rogue job or something)
Jarcec
> On Aug 21, 2015, at 9:18 AM, Andrew Bayer <an...@gmail.com> wrote:
>
> I'll take a look in 30 minutes or so.
> On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>
>> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
>> executors :) All the others are marked as “DEAD”.
>>
>> I’m wondering what is the process of reporting that? Should I create
>> BUILDS or INFRA JIRA, or just send email to this group or something
>> completely else?
>>
>> Jarcec
Re: What is the process of reporting dead slaves
Posted by Andrew Bayer <an...@gmail.com>.
I'll take a look in 30 minutes or so.
On Aug 21, 2015 12:05, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
> Looking at Jenkins queue - there is 132 queued jobs and only 4 working
> executors :) All the others are marked as “DEAD”.
>
> I’m wondering what is the process of reporting that? Should I create
> BUILDS or INFRA JIRA, or just send email to this group or something
> completely else?
>
> Jarcec