You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dong Joon Hyun <dh...@hortonworks.com> on 2017/08/21 00:45:27 UTC
Increase Timeout or optimize Spark UT?
Hi, All.
Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has been hitting the build timeout.
Please see the build time trend.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
All recent 22 builds fail due to timeout directly/indirectly. The last success (SBT with Hadoop-2.7) is 15th August.
We may do the followings.
1. Increase Build Timeout (3 hr 30 min)
2. Optimize UTs (Scala/Java/Python/UT)
But, Option 1 will be the immediate solution for now . Could you update the Jenkins setup?
Bests,
Dongjoon.
Re: Increase Timeout or optimize Spark UT?
Posted by Dongjoon Hyun <do...@gmail.com>.
BTW, the situation seems to become worse, now we lost two builds.
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/
Dongjoon.
On Thu, Aug 24, 2017 at 2:27 AM, Maciej Szymkiewicz <ms...@gmail.com>
wrote:
> It won't be used by PySpark and SparkR, will it?
>
> On 23 August 2017 at 23:40, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> I think we already set the number of partitions to 5 in tests
>> <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61>
>> ?
>>
>> On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz <
>> mszymkiewicz@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> From my experience it is possible to cut quite a lot by reducing
>>> spark.sql.shuffle.partitions to some reasonable value (let's say
>>> comparable to the number of cores). 200 is a serious overkill for most of
>>> the test cases anyway.
>>>
>>>
>>> Best,
>>> Maciej
>>>
>>>
>>>
>>> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com>
>>> wrote:
>>>
>>>> +1 for any efforts to recover Jenkins!
>>>>
>>>>
>>>>
>>>> Thank you for the direction.
>>>>
>>>>
>>>>
>>>> Bests,
>>>>
>>>> Dongjoon.
>>>>
>>>>
>>>>
>>>> *From: *Reynold Xin <rx...@databricks.com>
>>>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>>>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>>>> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
>>>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>>>
>>>>
>>>>
>>>> It seems like it's time to look into how to cut down some of the test
>>>> runtimes. Test runtimes will slowly go up given the way development
>>>> happens. 3 hr is already a very long time for tests to run.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>>>> wrote:
>>>>
>>>> Hi, All.
>>>>
>>>>
>>>>
>>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6)
>>>> has been hitting the build timeout.
>>>>
>>>>
>>>>
>>>> Please see the build time trend.
>>>>
>>>>
>>>>
>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>>>
>>>>
>>>>
>>>> All recent 22 builds fail due to timeout directly/indirectly. The last
>>>> success (SBT with Hadoop-2.7) is 15th August.
>>>>
>>>>
>>>>
>>>> We may do the followings.
>>>>
>>>>
>>>>
>>>> 1. Increase Build Timeout (3 hr 30 min)
>>>> 2. Optimize UTs (Scala/Java/Python/UT)
>>>>
>>>>
>>>>
>>>> But, Option 1 will be the immediate solution for now . Could you update
>>>> the Jenkins setup?
>>>>
>>>>
>>>>
>>>> Bests,
>>>>
>>>> Dongjoon.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
> --
>
> Z poważaniem,
> Maciej Szymkiewicz
>
>
Re: Increase Timeout or optimize Spark UT?
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
It won't be used by PySpark and SparkR, will it?
On 23 August 2017 at 23:40, Michael Armbrust <mi...@databricks.com> wrote:
> I think we already set the number of partitions to 5 in tests
> <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61>
> ?
>
> On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz <
> mszymkiewicz@gmail.com> wrote:
>
>> Hi,
>>
>> From my experience it is possible to cut quite a lot by reducing
>> spark.sql.shuffle.partitions to some reasonable value (let's say
>> comparable to the number of cores). 200 is a serious overkill for most of
>> the test cases anyway.
>>
>>
>> Best,
>> Maciej
>>
>>
>>
>> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
>>
>>> +1 for any efforts to recover Jenkins!
>>>
>>>
>>>
>>> Thank you for the direction.
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> *From: *Reynold Xin <rx...@databricks.com>
>>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>>> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
>>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>>
>>>
>>>
>>> It seems like it's time to look into how to cut down some of the test
>>> runtimes. Test runtimes will slowly go up given the way development
>>> happens. 3 hr is already a very long time for tests to run.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>>> wrote:
>>>
>>> Hi, All.
>>>
>>>
>>>
>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6)
>>> has been hitting the build timeout.
>>>
>>>
>>>
>>> Please see the build time trend.
>>>
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>>
>>>
>>>
>>> All recent 22 builds fail due to timeout directly/indirectly. The last
>>> success (SBT with Hadoop-2.7) is 15th August.
>>>
>>>
>>>
>>> We may do the followings.
>>>
>>>
>>>
>>> 1. Increase Build Timeout (3 hr 30 min)
>>> 2. Optimize UTs (Scala/Java/Python/UT)
>>>
>>>
>>>
>>> But, Option 1 will be the immediate solution for now . Could you update
>>> the Jenkins setup?
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>
>>
>
--
Z poważaniem,
Maciej Szymkiewicz
Re: Increase Timeout or optimize Spark UT?
Posted by Michael Armbrust <mi...@databricks.com>.
I think we already set the number of partitions to 5 in tests
<https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61>
?
On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz <ms...@gmail.com>
wrote:
> Hi,
>
> From my experience it is possible to cut quite a lot by reducing
> spark.sql.shuffle.partitions to some reasonable value (let's say
> comparable to the number of cores). 200 is a serious overkill for most of
> the test cases anyway.
>
>
> Best,
> Maciej
>
>
>
> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
>
>> +1 for any efforts to recover Jenkins!
>>
>>
>>
>> Thank you for the direction.
>>
>>
>>
>> Bests,
>>
>> Dongjoon.
>>
>>
>>
>> *From: *Reynold Xin <rx...@databricks.com>
>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>
>>
>>
>> It seems like it's time to look into how to cut down some of the test
>> runtimes. Test runtimes will slowly go up given the way development
>> happens. 3 hr is already a very long time for tests to run.
>>
>>
>>
>>
>>
>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>> wrote:
>>
>> Hi, All.
>>
>>
>>
>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has
>> been hitting the build timeout.
>>
>>
>>
>> Please see the build time trend.
>>
>>
>>
>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>
>>
>>
>> All recent 22 builds fail due to timeout directly/indirectly. The last
>> success (SBT with Hadoop-2.7) is 15th August.
>>
>>
>>
>> We may do the followings.
>>
>>
>>
>> 1. Increase Build Timeout (3 hr 30 min)
>> 2. Optimize UTs (Scala/Java/Python/UT)
>>
>>
>>
>> But, Option 1 will be the immediate solution for now . Could you update
>> the Jenkins setup?
>>
>>
>>
>> Bests,
>>
>> Dongjoon.
>>
>>
>>
>
>
Re: Increase Timeout or optimize Spark UT?
Posted by Mark Hamstra <ma...@clearstorydata.com>.
This is another argument for getting the code to the point where this can
default to "true":
SQLConf.scala: val ADAPTIVE_EXECUTION_ENABLED = buildConf("
*spark.sql.adaptive.enabled*")
On Tue, Aug 22, 2017 at 12:27 PM, Reynold Xin <rx...@databricks.com> wrote:
> +1
>
>
> On Tue, Aug 22, 2017 at 12:25 PM, Maciej Szymkiewicz <
> mszymkiewicz@gmail.com> wrote:
>
>> Hi,
>>
>> From my experience it is possible to cut quite a lot by reducing
>> spark.sql.shuffle.partitions to some reasonable value (let's say
>> comparable to the number of cores). 200 is a serious overkill for most of
>> the test cases anyway.
>>
>>
>> Best,
>> Maciej
>>
>>
>>
>> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
>>
>>> +1 for any efforts to recover Jenkins!
>>>
>>>
>>>
>>> Thank you for the direction.
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> *From: *Reynold Xin <rx...@databricks.com>
>>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>>> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
>>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>>
>>>
>>>
>>> It seems like it's time to look into how to cut down some of the test
>>> runtimes. Test runtimes will slowly go up given the way development
>>> happens. 3 hr is already a very long time for tests to run.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>>> wrote:
>>>
>>> Hi, All.
>>>
>>>
>>>
>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6)
>>> has been hitting the build timeout.
>>>
>>>
>>>
>>> Please see the build time trend.
>>>
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>>
>>>
>>>
>>> All recent 22 builds fail due to timeout directly/indirectly. The last
>>> success (SBT with Hadoop-2.7) is 15th August.
>>>
>>>
>>>
>>> We may do the followings.
>>>
>>>
>>>
>>> 1. Increase Build Timeout (3 hr 30 min)
>>> 2. Optimize UTs (Scala/Java/Python/UT)
>>>
>>>
>>>
>>> But, Option 1 will be the immediate solution for now . Could you update
>>> the Jenkins setup?
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>
>>
>
Re: Increase Timeout or optimize Spark UT?
Posted by Reynold Xin <rx...@databricks.com>.
+1
On Tue, Aug 22, 2017 at 12:25 PM, Maciej Szymkiewicz <mszymkiewicz@gmail.com
> wrote:
> Hi,
>
> From my experience it is possible to cut quite a lot by reducing
> spark.sql.shuffle.partitions to some reasonable value (let's say
> comparable to the number of cores). 200 is a serious overkill for most of
> the test cases anyway.
>
>
> Best,
> Maciej
>
>
>
> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
>
>> +1 for any efforts to recover Jenkins!
>>
>>
>>
>> Thank you for the direction.
>>
>>
>>
>> Bests,
>>
>> Dongjoon.
>>
>>
>>
>> *From: *Reynold Xin <rx...@databricks.com>
>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>
>>
>>
>> It seems like it's time to look into how to cut down some of the test
>> runtimes. Test runtimes will slowly go up given the way development
>> happens. 3 hr is already a very long time for tests to run.
>>
>>
>>
>>
>>
>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>> wrote:
>>
>> Hi, All.
>>
>>
>>
>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has
>> been hitting the build timeout.
>>
>>
>>
>> Please see the build time trend.
>>
>>
>>
>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>
>>
>>
>> All recent 22 builds fail due to timeout directly/indirectly. The last
>> success (SBT with Hadoop-2.7) is 15th August.
>>
>>
>>
>> We may do the followings.
>>
>>
>>
>> 1. Increase Build Timeout (3 hr 30 min)
>> 2. Optimize UTs (Scala/Java/Python/UT)
>>
>>
>>
>> But, Option 1 will be the immediate solution for now . Could you update
>> the Jenkins setup?
>>
>>
>>
>> Bests,
>>
>> Dongjoon.
>>
>>
>>
>
>
Re: Increase Timeout or optimize Spark UT?
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
Hi,
From my experience it is possible to cut quite a lot by reducing
spark.sql.shuffle.partitions to some reasonable value (let's say comparable
to the number of cores). 200 is a serious overkill for most of the test
cases anyway.
Best,
Maciej
On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
> +1 for any efforts to recover Jenkins!
>
>
>
> Thank you for the direction.
>
>
>
> Bests,
>
> Dongjoon.
>
>
>
> *From: *Reynold Xin <rx...@databricks.com>
> *Date: *Sunday, August 20, 2017 at 5:53 PM
> *To: *Dong Joon Hyun <dh...@hortonworks.com>
> *Cc: *"dev@spark.apache.org" <de...@spark.apache.org>
> *Subject: *Re: Increase Timeout or optimize Spark UT?
>
>
>
> It seems like it's time to look into how to cut down some of the test
> runtimes. Test runtimes will slowly go up given the way development
> happens. 3 hr is already a very long time for tests to run.
>
>
>
>
>
> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
> wrote:
>
> Hi, All.
>
>
>
> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has
> been hitting the build timeout.
>
>
>
> Please see the build time trend.
>
>
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%
> 20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>
>
>
> All recent 22 builds fail due to timeout directly/indirectly. The last
> success (SBT with Hadoop-2.7) is 15th August.
>
>
>
> We may do the followings.
>
>
>
> 1. Increase Build Timeout (3 hr 30 min)
> 2. Optimize UTs (Scala/Java/Python/UT)
>
>
>
> But, Option 1 will be the immediate solution for now . Could you update
> the Jenkins setup?
>
>
>
> Bests,
>
> Dongjoon.
>
>
>
Re: Increase Timeout or optimize Spark UT?
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
Hi,
Re: Increase Timeout or optimize Spark UT?
Posted by Dong Joon Hyun <dh...@hortonworks.com>.
+1 for any efforts to recover Jenkins!
Thank you for the direction.
Bests,
Dongjoon.
From: Reynold Xin <rx...@databricks.com>
Date: Sunday, August 20, 2017 at 5:53 PM
To: Dong Joon Hyun <dh...@hortonworks.com>
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Subject: Re: Increase Timeout or optimize Spark UT?
It seems like it's time to look into how to cut down some of the test runtimes. Test runtimes will slowly go up given the way development happens. 3 hr is already a very long time for tests to run.
On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>> wrote:
Hi, All.
Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has been hitting the build timeout.
Please see the build time trend.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
All recent 22 builds fail due to timeout directly/indirectly. The last success (SBT with Hadoop-2.7) is 15th August.
We may do the followings.
1. Increase Build Timeout (3 hr 30 min)
2. Optimize UTs (Scala/Java/Python/UT)
But, Option 1 will be the immediate solution for now . Could you update the Jenkins setup?
Bests,
Dongjoon.
Re: Increase Timeout or optimize Spark UT?
Posted by Reynold Xin <rx...@databricks.com>.
It seems like it's time to look into how to cut down some of the test
runtimes. Test runtimes will slowly go up given the way development
happens. 3 hr is already a very long time for tests to run.
On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
wrote:
> Hi, All.
>
>
>
> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has
> been hitting the build timeout.
>
>
>
> Please see the build time trend.
>
>
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%
> 20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>
>
>
> All recent 22 builds fail due to timeout directly/indirectly. The last
> success (SBT with Hadoop-2.7) is 15th August.
>
>
>
> We may do the followings.
>
>
>
> 1. Increase Build Timeout (3 hr 30 min)
> 2. Optimize UTs (Scala/Java/Python/UT)
>
>
>
> But, Option 1 will be the immediate solution for now . Could you update
> the Jenkins setup?
>
>
>
> Bests,
>
> Dongjoon.
>