You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dilipbiswal <gi...@git.apache.org> on 2018/10/31 06:31:06 UTC
[GitHub] spark pull request #22900: [SPARK-25618][SQL][TEST] Reduce time taken to exe...
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/22900
[SPARK-25618][SQL][TEST] Reduce time taken to execute KafkaContinuousSourceStressForDontFailOnDataLossSuite
## What changes were proposed in this pull request?
In this test, i have reduced the test time to 20 secs from 1 minute while reducing the sleep time from 1 sec to 100 milliseconds.
With this change, i was able to run the test in 20+ seconds consistently on my laptop. I would like see if it passes in jenkins consistently.
## How was this patch tested?
Its a test fix.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark SPARK-25618
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22900.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22900
----
commit 5a6907cd1dc0872d68214bc42f1b2898f9e48009
Author: Dilip Biswal <db...@...>
Date: 2018-10-17T05:08:19Z
[SPARK-25618] Reduce time taken to execute KafkaContinuousSourceStressForDontFailOnDataLossSuite
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22900: [SPARK-25618][SQL][TEST] Reduce time taken to exe...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22900#discussion_r229619816
--- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala ---
@@ -221,7 +221,7 @@ class KafkaSourceStressForDontFailOnDataLossSuite extends StreamTest with KafkaM
.as[(String, String)]
val query = startStream(kafka.map(kv => kv._2.toInt))
- val testTime = 1.minutes
+ val testTime = 20.seconds
--- End diff --
@dilipbiswal . nit. Although this is a random stress test, can we compare the numbers of Kafka operation of before and after in general? Although the test time is reduced to 1/3, line 255 seems to recover it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4650/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22900
**[Test build #98300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98300/testReport)** for PR 22900 at commit [`5a6907c`](https://github.com/apache/spark/commit/5a6907cd1dc0872d68214bc42f1b2898f9e48009).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by dilipbiswal <gi...@git.apache.org>.
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/22900
cc @srowen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22900
Merged to master. Thank you, @dilipbiswal .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by dilipbiswal <gi...@git.apache.org>.
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/22900
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98300/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98296/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4652/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by dilipbiswal <gi...@git.apache.org>.
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/22900
Oh...thank you very much @dongjoon-hyun
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22900
**[Test build #98296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98296/testReport)** for PR 22900 at commit [`5a6907c`](https://github.com/apache/spark/commit/5a6907cd1dc0872d68214bc42f1b2898f9e48009).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22900: [SPARK-25618][SQL][TEST] Reduce time taken to exe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22900
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22900
**[Test build #98300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98300/testReport)** for PR 22900 at commit [`5a6907c`](https://github.com/apache/spark/commit/5a6907cd1dc0872d68214bc42f1b2898f9e48009).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22900: [SPARK-25618][SQL][TEST] Reduce time taken to exe...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22900#discussion_r229626535
--- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala ---
@@ -221,7 +221,7 @@ class KafkaSourceStressForDontFailOnDataLossSuite extends StreamTest with KafkaM
.as[(String, String)]
val query = startStream(kafka.map(kv => kv._2.toInt))
- val testTime = 1.minutes
+ val testTime = 20.seconds
--- End diff --
Thank you for confirmation. Then, it's okay.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22900
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22900: [SPARK-25618][SQL][TEST] Reduce time taken to execute Ka...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22900
**[Test build #98296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98296/testReport)** for PR 22900 at commit [`5a6907c`](https://github.com/apache/spark/commit/5a6907cd1dc0872d68214bc42f1b2898f9e48009).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22900: [SPARK-25618][SQL][TEST] Reduce time taken to exe...
Posted by dilipbiswal <gi...@git.apache.org>.
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/22900#discussion_r229625779
--- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala ---
@@ -221,7 +221,7 @@ class KafkaSourceStressForDontFailOnDataLossSuite extends StreamTest with KafkaM
.as[(String, String)]
val query = startStream(kafka.map(kv => kv._2.toInt))
- val testTime = 1.minutes
+ val testTime = 20.seconds
--- End diff --
@dongjoon-hyun Actually i did count the number of times we went inside each case block. With this change, we did a little less than we used to do before.. but if i remember, we did enough kafka operations. I will get the count of each case block tomorrow for your perusal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org