You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2020/05/28 08:13:19 UTC

Build time limit in PR builder

Hi all,

Seems we're hitting the time limit in PR builders (see
https://github.com/apache/spark/pull/28627), in particular wen it's Maven
build which takes more time compared to SBT in general.
Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to
unblock these PRs and focus on cutting the build/testing time?

Re: Build time limit in PR builder

Posted by shane knapp ☠ <sk...@berkeley.edu>.
the timer is set to 500m now in master, 3.0 and 2.4.

On Thu, May 28, 2020 at 12:32 PM Kousuke Saruta <sa...@oss.nttdata.com>
wrote:

> Thanks all. It's very helpful!
>
> - Kousuke
>
> On 2020/05/29 3:31, shane knapp ☠ wrote:
>
> https://github.com/apache/spark/pull/28666
>
> On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sk...@berkeley.edu>
> wrote:
>
>> i'll get a PR put together now.
>>
>> On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> I remember we were able to cut down pretty considerably in the past. For
>>> example, I investigated (
>>> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
>>> and fixed some before at, like
>>> https://github.com/apache/spark/pull/23111. Maybe we could skim again
>>> to reduce the build/testing time ..
>>>
>>> But I agree with increasing it at this moment for now to unblock PRs.
>>> Don't forget to fix here too
>>> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
>>>  :-).
>>>
>>> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>>>
>>>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> What else can we do, I suppose?
>>>>>
>>>>> there,s not much else we can do.  i'll add 30m to the timeout.
>>>>
>>>> shane
>>>> --
>>>> Shane Knapp
>>>> Computer Guy / Voice of Reason
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
>
>

-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Build time limit in PR builder

Posted by Kousuke Saruta <sa...@oss.nttdata.com>.
Thanks all. It's very helpful!

- Kousuke

On 2020/05/29 3:31, shane knapp ☠ wrote:
> https://github.com/apache/spark/pull/28666
>
> On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sknapp@berkeley.edu <ma...@berkeley.edu>> wrote:
>
>     i'll get a PR put together now.
>
>     On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
>
>         I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
>         and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..
>
>         But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).
>
>         2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sknapp@berkeley.edu <ma...@berkeley.edu>>님이 작성:
>
>             On Thu, May 28, 2020 at 7:16 AM Sean Owen <srowen@gmail.com <ma...@gmail.com>> wrote:
>
>                 What else can we do, I suppose?
>
>             there,s not much else we can do.  i'll add 30m to the timeout.
>
>             shane
>             -- 
>             Shane Knapp
>             Computer Guy / Voice of Reason
>             UC Berkeley EECS Research / RISELab Staff Technical Lead
>             https://rise.cs.berkeley.edu
>
>
>
>     -- 
>     Shane Knapp
>     Computer Guy / Voice of Reason
>     UC Berkeley EECS Research / RISELab Staff Technical Lead
>     https://rise.cs.berkeley.edu
>
>
>
> -- 
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


Re: Build time limit in PR builder

Posted by shane knapp ☠ <sk...@berkeley.edu>.
https://github.com/apache/spark/pull/28666

On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sk...@berkeley.edu> wrote:

> i'll get a PR put together now.
>
> On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> I remember we were able to cut down pretty considerably in the past. For
>> example, I investigated (
>> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
>> and fixed some before at, like https://github.com/apache/spark/pull/23111.
>> Maybe we could skim again to reduce the build/testing time ..
>>
>> But I agree with increasing it at this moment for now to unblock PRs.
>> Don't forget to fix here too
>> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
>>  :-).
>>
>> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>>
>>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> What else can we do, I suppose?
>>>>
>>>> there,s not much else we can do.  i'll add 30m to the timeout.
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> Computer Guy / Voice of Reason
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>


-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Build time limit in PR builder

Posted by shane knapp ☠ <sk...@berkeley.edu>.
i'll get a PR put together now.

On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:

> I remember we were able to cut down pretty considerably in the past. For
> example, I investigated (
> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
> and fixed some before at, like https://github.com/apache/spark/pull/23111.
> Maybe we could skim again to reduce the build/testing time ..
>
> But I agree with increasing it at this moment for now to unblock PRs.
> Don't forget to fix here too
> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
>  :-).
>
> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>
>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>
>>> What else can we do, I suppose?
>>>
>>> there,s not much else we can do.  i'll add 30m to the timeout.
>>
>> shane
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>

-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Build time limit in PR builder

Posted by Hyukjin Kwon <gu...@gmail.com>.
I remember we were able to cut down pretty considerably in the past. For
example, I investigated (
https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111.
Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't
forget to fix here too
https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:

> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>
>> What else can we do, I suppose?
>>
>> there,s not much else we can do.  i'll add 30m to the timeout.
>
> shane
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>

Re: Build time limit in PR builder

Posted by shane knapp ☠ <sk...@berkeley.edu>.
On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:

> What else can we do, I suppose?
>
> there,s not much else we can do.  i'll add 30m to the timeout.

shane
-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Build time limit in PR builder

Posted by Sean Owen <sr...@gmail.com>.
What else can we do, I suppose?

On Thu, May 28, 2020 at 3:13 AM Hyukjin Kwon <gu...@gmail.com> wrote:

> Hi all,
>
> Seems we're hitting the time limit in PR builders (see
> https://github.com/apache/spark/pull/28627), in particular wen it's Maven
> build which takes more time compared to SBT in general.
> Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to
> unblock these PRs and focus on cutting the build/testing time?
>
>