You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp ☠ <sk...@berkeley.edu> on 2021/08/09 16:17:05 UTC
[build system] half of the jenkins workers are down
happy monday!
the server gods did not smile upon us this weekend, and 4 of the workers
are down. we'll most likely need to head to our colo some time today and
give them an in-person kick and see what's going on.
i'll send an update when they're back up.
shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [build system] half of the jenkins workers are down
Posted by Xiao Li <ga...@gmail.com>.
Thank you, Shane!
Xiao
shane knapp ☠ <sk...@berkeley.edu> 于2021年8月9日周一 下午1:26写道：
> turns out that minikube/k8s and friends were being oom-killed and this was
> causing all sorts of weirdnesses.
>
> i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and
> we'll keep an eye on things and see how they go.
>
> On Mon, Aug 9, 2021 at 12:02 PM shane knapp ☠ <sk...@berkeley.edu> wrote:
>
>> as workers are continuing to fail, i've stopped jenkins from accepting
>> new builds for the time being.
>>
>> more updates as they come.
>>
>> On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ <sk...@berkeley.edu> wrote:
>>
>>> happy monday!
>>>
>>> the server gods did not smile upon us this weekend, and 4 of the workers
>>> are down. we'll most likely need to head to our colo some time today and
>>> give them an in-person kick and see what's going on.
>>>
>>> i'll send an update when they're back up.
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> Computer Guy / Voice of Reason
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: [build system] half of the jenkins workers are down
Posted by shane knapp ☠ <sk...@berkeley.edu>.
turns out that minikube/k8s and friends were being oom-killed and this was
causing all sorts of weirdnesses.
i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and we'll
keep an eye on things and see how they go.
On Mon, Aug 9, 2021 at 12:02 PM shane knapp ☠ <sk...@berkeley.edu> wrote:
> as workers are continuing to fail, i've stopped jenkins from accepting new
> builds for the time being.
>
> more updates as they come.
>
> On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ <sk...@berkeley.edu> wrote:
>
>> happy monday!
>>
>> the server gods did not smile upon us this weekend, and 4 of the workers
>> are down. we'll most likely need to head to our colo some time today and
>> give them an in-person kick and see what's going on.
>>
>> i'll send an update when they're back up.
>>
>> shane
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [build system] half of the jenkins workers are down
Posted by shane knapp ☠ <sk...@berkeley.edu>.
as workers are continuing to fail, i've stopped jenkins from accepting new
builds for the time being.
more updates as they come.
On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ <sk...@berkeley.edu> wrote:
> happy monday!
>
> the server gods did not smile upon us this weekend, and 4 of the workers
> are down. we'll most likely need to head to our colo some time today and
> give them an in-person kick and see what's going on.
>
> i'll send an update when they're back up.
>
> shane
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu