You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2020/05/28 08:13:19 UTC
Build time limit in PR builder
Hi all,
Seems we're hitting the time limit in PR builders (see
https://github.com/apache/spark/pull/28627), in particular wen it's Maven
build which takes more time compared to SBT in general.
Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to
unblock these PRs and focus on cutting the build/testing time?
Re: Build time limit in PR builder
Posted by shane knapp ☠ <sk...@berkeley.edu>.
the timer is set to 500m now in master, 3.0 and 2.4.
On Thu, May 28, 2020 at 12:32 PM Kousuke Saruta <sa...@oss.nttdata.com>
wrote:
> Thanks all. It's very helpful!
>
> - Kousuke
>
> On 2020/05/29 3:31, shane knapp ☠ wrote:
>
> https://github.com/apache/spark/pull/28666
>
> On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sk...@berkeley.edu>
> wrote:
>
>> i'll get a PR put together now.
>>
>> On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> I remember we were able to cut down pretty considerably in the past. For
>>> example, I investigated (
>>> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
>>> and fixed some before at, like
>>> https://github.com/apache/spark/pull/23111. Maybe we could skim again
>>> to reduce the build/testing time ..
>>>
>>> But I agree with increasing it at this moment for now to unblock PRs.
>>> Don't forget to fix here too
>>> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
>>> :-).
>>>
>>> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>>>
>>>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> What else can we do, I suppose?
>>>>>
>>>>> there,s not much else we can do. i'll add 30m to the timeout.
>>>>
>>>> shane
>>>> --
>>>> Shane Knapp
>>>> Computer Guy / Voice of Reason
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
>
>
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Build time limit in PR builder
Posted by Kousuke Saruta <sa...@oss.nttdata.com>.
Thanks all. It's very helpful!
- Kousuke
On 2020/05/29 3:31, shane knapp ☠ wrote:
> https://github.com/apache/spark/pull/28666
>
> On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sknapp@berkeley.edu <ma...@berkeley.edu>> wrote:
>
> i'll get a PR put together now.
>
> On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
>
> I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
> and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..
>
> But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).
>
> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sknapp@berkeley.edu <ma...@berkeley.edu>>님이 작성:
>
> On Thu, May 28, 2020 at 7:16 AM Sean Owen <srowen@gmail.com <ma...@gmail.com>> wrote:
>
> What else can we do, I suppose?
>
> there,s not much else we can do. i'll add 30m to the timeout.
>
> shane
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
Re: Build time limit in PR builder
Posted by shane knapp ☠ <sk...@berkeley.edu>.
https://github.com/apache/spark/pull/28666
On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <sk...@berkeley.edu> wrote:
> i'll get a PR put together now.
>
> On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> I remember we were able to cut down pretty considerably in the past. For
>> example, I investigated (
>> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
>> and fixed some before at, like https://github.com/apache/spark/pull/23111.
>> Maybe we could skim again to reduce the build/testing time ..
>>
>> But I agree with increasing it at this moment for now to unblock PRs.
>> Don't forget to fix here too
>> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
>> :-).
>>
>> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>>
>>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> What else can we do, I suppose?
>>>>
>>>> there,s not much else we can do. i'll add 30m to the timeout.
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> Computer Guy / Voice of Reason
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Build time limit in PR builder
Posted by shane knapp ☠ <sk...@berkeley.edu>.
i'll get a PR put together now.
On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> I remember we were able to cut down pretty considerably in the past. For
> example, I investigated (
> https://github.com/apache/spark/pull/21822#issuecomment-407295739)
> and fixed some before at, like https://github.com/apache/spark/pull/23111.
> Maybe we could skim again to reduce the build/testing time ..
>
> But I agree with increasing it at this moment for now to unblock PRs.
> Don't forget to fix here too
> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
> :-).
>
> 2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
>
>> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>>
>>> What else can we do, I suppose?
>>>
>>> there,s not much else we can do. i'll add 30m to the timeout.
>>
>> shane
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Build time limit in PR builder
Posted by Hyukjin Kwon <gu...@gmail.com>.
I remember we were able to cut down pretty considerably in the past. For
example, I investigated (
https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111.
Maybe we could skim again to reduce the build/testing time ..
But I agree with increasing it at this moment for now to unblock PRs. Don't
forget to fix here too
https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201
:-).
2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <sk...@berkeley.edu>님이 작성:
> On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
>
>> What else can we do, I suppose?
>>
>> there,s not much else we can do. i'll add 30m to the timeout.
>
> shane
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: Build time limit in PR builder
Posted by shane knapp ☠ <sk...@berkeley.edu>.
On Thu, May 28, 2020 at 7:16 AM Sean Owen <sr...@gmail.com> wrote:
> What else can we do, I suppose?
>
> there,s not much else we can do. i'll add 30m to the timeout.
shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Build time limit in PR builder
Posted by Sean Owen <sr...@gmail.com>.
What else can we do, I suppose?
On Thu, May 28, 2020 at 3:13 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> Hi all,
>
> Seems we're hitting the time limit in PR builders (see
> https://github.com/apache/spark/pull/28627), in particular wen it's Maven
> build which takes more time compared to SBT in general.
> Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to
> unblock these PRs and focus on cutting the build/testing time?
>
>