You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2021/07/20 10:30:20 UTC

Flaky build in GitHub Actions

Hi all,

Looks like there's something going on in the machines in GitHub Actions.
The build is now very flaky and keeps dying with symptoms like I guess
out-of-memory (?).
I will try to take a closer look tomorrow but it would be great if you guys
find some time to take a look into it 🙏

Re: Flaky build in GitHub Actions

Posted by Xiao Li <li...@databricks.com>.

Thank you, Liang-chi and Hyukjin!

On Sun, Jul 25, 2021 at 6:25 PM Hyukjin Kwon <gu...@gmail.com> wrote:

> This is fixed up via Laingchi's PR:
> https://github.com/apache/spark/pull/33447. The issue is almost fixed now
> and less flaky.
> I'm still interacting w/ GitHub Actions: they are still investigating the
> issue. Seems like there's no similar ticket reported so they suspect an
> issue specific to Apahc Spark repo.
>
>
> 2021년 7월 22일 (목) 오전 9:40, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> FYI, @Liang-Chi Hsieh <vi...@gmail.com> is trying to control the memory
>> in the test base at https://github.com/apache/spark/pull/33447 which
>> looks almost promising now.
>> While I don't object to merge things, would need to closely track how
>> these tests go at Github Actions in his PR (and in the main Apache repo)
>>
>> 2021년 7월 22일 (목) 오전 3:00, Holden Karau <ho...@pigscanfly.ca>님이 작성:
>>
>>> I noticed that the worker decommissioning suite maybe seems to be
>>> running up against the memory limits so I'm going to try and see if I can
>>> get our memory usage down a bit as well while we wait for GH response. In
>>> the meantime, I'm assuming if things pass Jenkins we are OK with merging
>>> yes?
>>>
>>> On Wed, Jul 21, 2021 at 10:03 AM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Thank you, Hyukjin!
>>>>
>>>> Dongjoon.
>>>>
>>>> On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> I filed a ticket at GitHub. I will share more details when I get a
>>>>> response from them.
>>>>>
>>>>> 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Looks like there's something going on in the machines in GitHub
>>>>>> Actions.
>>>>>> The build is now very flaky and keeps dying with symptoms like I
>>>>>> guess out-of-memory (?).
>>>>>> I will try to take a closer look tomorrow but it would be great if
>>>>>> you guys find some time to take a look into it 🙏
>>>>>>
>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

--

Re: Flaky build in GitHub Actions

Posted by Hyukjin Kwon <gu...@gmail.com>.

This is fixed up via Laingchi's PR:
https://github.com/apache/spark/pull/33447. The issue is almost fixed now
and less flaky.
I'm still interacting w/ GitHub Actions: they are still investigating the
issue. Seems like there's no similar ticket reported so they suspect an
issue specific to Apahc Spark repo.


2021년 7월 22일 (목) 오전 9:40, Hyukjin Kwon <gu...@gmail.com>님이 작성:

> FYI, @Liang-Chi Hsieh <vi...@gmail.com> is trying to control the memory
> in the test base at https://github.com/apache/spark/pull/33447 which
> looks almost promising now.
> While I don't object to merge things, would need to closely track how
> these tests go at Github Actions in his PR (and in the main Apache repo)
>
> 2021년 7월 22일 (목) 오전 3:00, Holden Karau <ho...@pigscanfly.ca>님이 작성:
>
>> I noticed that the worker decommissioning suite maybe seems to be running
>> up against the memory limits so I'm going to try and see if I can get our
>> memory usage down a bit as well while we wait for GH response. In the
>> meantime, I'm assuming if things pass Jenkins we are OK with merging yes?
>>
>> On Wed, Jul 21, 2021 at 10:03 AM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Thank you, Hyukjin!
>>>
>>> Dongjoon.
>>>
>>> On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> I filed a ticket at GitHub. I will share more details when I get a
>>>> response from them.
>>>>
>>>> 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Looks like there's something going on in the machines in GitHub
>>>>> Actions.
>>>>> The build is now very flaky and keeps dying with symptoms like I guess
>>>>> out-of-memory (?).
>>>>> I will try to take a closer look tomorrow but it would be great if you
>>>>> guys find some time to take a look into it 🙏
>>>>>
>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Flaky build in GitHub Actions

Posted by Hyukjin Kwon <gu...@gmail.com>.

FYI, @Liang-Chi Hsieh <vi...@gmail.com> is trying to control the memory in
the test base at https://github.com/apache/spark/pull/33447 which looks
almost promising now.
While I don't object to merge things, would need to closely track how these
tests go at Github Actions in his PR (and in the main Apache repo)

2021년 7월 22일 (목) 오전 3:00, Holden Karau <ho...@pigscanfly.ca>님이 작성:

> I noticed that the worker decommissioning suite maybe seems to be running
> up against the memory limits so I'm going to try and see if I can get our
> memory usage down a bit as well while we wait for GH response. In the
> meantime, I'm assuming if things pass Jenkins we are OK with merging yes?
>
> On Wed, Jul 21, 2021 at 10:03 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Thank you, Hyukjin!
>>
>> Dongjoon.
>>
>> On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> I filed a ticket at GitHub. I will share more details when I get a
>>> response from them.
>>>
>>> 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> Hi all,
>>>>
>>>> Looks like there's something going on in the machines in GitHub Actions.
>>>> The build is now very flaky and keeps dying with symptoms like I guess
>>>> out-of-memory (?).
>>>> I will try to take a closer look tomorrow but it would be great if you
>>>> guys find some time to take a look into it 🙏
>>>>
>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Flaky build in GitHub Actions

Posted by Holden Karau <ho...@pigscanfly.ca>.

I noticed that the worker decommissioning suite maybe seems to be running
up against the memory limits so I'm going to try and see if I can get our
memory usage down a bit as well while we wait for GH response. In the
meantime, I'm assuming if things pass Jenkins we are OK with merging yes?

On Wed, Jul 21, 2021 at 10:03 AM Dongjoon Hyun <do...@gmail.com>
wrote:

> Thank you, Hyukjin!
>
> Dongjoon.
>
> On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> I filed a ticket at GitHub. I will share more details when I get a
>> response from them.
>>
>> 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>
>>> Hi all,
>>>
>>> Looks like there's something going on in the machines in GitHub Actions.
>>> The build is now very flaky and keeps dying with symptoms like I guess
>>> out-of-memory (?).
>>> I will try to take a closer look tomorrow but it would be great if you
>>> guys find some time to take a look into it 🙏
>>>
>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Flaky build in GitHub Actions

Posted by Dongjoon Hyun <do...@gmail.com>.

Thank you, Hyukjin!

Dongjoon.

On Tue, Jul 20, 2021 at 8:53 PM Hyukjin Kwon <gu...@gmail.com> wrote:

> I filed a ticket at GitHub. I will share more details when I get a
> response from them.
>
> 2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> Hi all,
>>
>> Looks like there's something going on in the machines in GitHub Actions.
>> The build is now very flaky and keeps dying with symptoms like I guess
>> out-of-memory (?).
>> I will try to take a closer look tomorrow but it would be great if you
>> guys find some time to take a look into it 🙏
>>
>

Re: Flaky build in GitHub Actions

Posted by Hyukjin Kwon <gu...@gmail.com>.

I filed a ticket at GitHub. I will share more details when I get a response
from them.

2021년 7월 20일 (화) 오후 7:30, Hyukjin Kwon <gu...@gmail.com>님이 작성:

> Hi all,
>
> Looks like there's something going on in the machines in GitHub Actions.
> The build is now very flaky and keeps dying with symptoms like I guess
> out-of-memory (?).
> I will try to take a closer look tomorrow but it would be great if you
> guys find some time to take a look into it 🙏
>