You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/26 06:25:21 UTC
[GitHub] [spark] zhengruifeng opened a new pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
zhengruifeng opened a new pull request #34108:
URL: https://github.com/apache/spark/pull/34108
this is only used to check correctness for correctness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-1007033566
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927253743
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927242653
this local result should be like:
```
Query 1: answer = 37714
Query 1: splitting []: 37714
Query 1: splitting [7]: 37714
Query 1: splitting [6]: 37714
Query 1: splitting [6,7]: 37714
Query 1: splitting [3]: 37714
Query 1: splitting [3,7]: 37714
Query 1: splitting [3,6]: 37714
Query 1: splitting [3,6,7]: 37714
Query 1: splitting [0]: 37714
Query 1: splitting [0,7]: 37714
Query 1: splitting [0,6]: 37714
Query 1: splitting [0,6,7]: 37714
Query 1: splitting [0,3]: 37714
Query 1: splitting [0,3,7]: 37714
Query 1: splitting [0,3,6]: 37714
Query 1: splitting [0,3,6,7]: 37714
Query 1: random splits
Query 1: splitting []: 37714
Query 1: splitting [9]: 113142
Query 1: splitting [8]: 155044
Query 1: splitting [8,9]: 465132
Query 1: splitting [7]: 37714
Query 1: splitting [7,9]: 113142
Query 1: splitting [7,8]: 155044
Query 1: splitting [7,8,9]: 465132
Query 1: splitting [6]: 37714
Query 1: splitting [6,9]: 113142
Query 1: splitting [6,8]: 155044
Query 1: splitting [6,8,9]: 465132
Query 1: splitting [6,7]: 37714
[info] *** Test still running after 4 minutes, 58 seconds: suite name: GeneralSkewSuite, test name: Query 1: 10-Table Join.
Query 1: splitting [6,7,9]: 113142
Query 1: splitting [6,7,8]: 155044
Query 1: splitting [6,7,8,9]: 465132
Query 1: splitting [5]: 37714
Query 1: splitting [5,9]: 113142
Query 1: splitting [5,8]: 155044
Query 1: splitting [5,8,9]: 465132
[info] - Query 1: 10-Table Join (7 minutes, 25 seconds)
Query 2: answer = 324
Query 2: splitting []: 324
Query 2: splitting [6]: 324
Query 2: splitting [2]: 324
Query 2: splitting [2,6]: 324
Query 2: splitting [1]: 324
Query 2: splitting [1,6]: 324
Query 2: splitting [1,2]: 324
Query 2: splitting [1,2,6]: 324
Query 2: random splits
Query 2: splitting []: 324
Query 2: splitting [9]: 14266
Query 2: splitting [8]: 35146
Query 2: splitting [8,9]: 151222
Query 2: splitting [7]: 540
Query 2: splitting [7,9]: 17110
Query 2: splitting [7,8]: 42870
Query 2: splitting [7,8,9]: 179690
Query 2: splitting [6]: 324
Query 2: splitting [6,9]: 14266
Query 2: splitting [6,8]: 35146
Query 2: splitting [6,8,9]: 151222
Query 2: splitting [6,7]: 540
Query 2: splitting [6,7,9]: 17110
Query 2: splitting [6,7,8]: 42870
[info] *** Test still running after 2 minutes, 33 seconds: suite name: GeneralSkewSuite, test name: Query 2: 10-Table Join.
Query 2: splitting [6,7,8,9]: 179690
Query 2: splitting [5]: 540
Query 2: splitting [5,9]: 26895
Query 2: splitting [5,8]: 56775
Query 2: splitting [5,8,9]: 256425
[info] - Query 2: 10-Table Join (3 minutes, 56 seconds)
Query 3: answer = 38038
Query 3: splitting []: 38038
Query 3: splitting [5]: 38038
Query 3: splitting [4]: 38038
Query 3: splitting [4,5]: 38038
Query 3: splitting [2]: 38038
Query 3: splitting [2,5]: 38038
Query 3: splitting [2,4]: 38038
Query 3: splitting [2,4,5]: 38038
Query 3: random splits
Query 3: splitting []: 38038
Query 3: splitting [9]: 146604
Query 3: splitting [8]: 190190
Query 3: splitting [8,9]: 570570
Query 3: splitting [7]: 40930
Query 3: splitting [7,9]: 158380
Query 3: splitting [7,8]: 204650
Query 3: splitting [7,8,9]: 613950
Query 3: splitting [6]: 38038
Query 3: splitting [6,9]: 146604
Query 3: splitting [6,8]: 190190
Query 3: splitting [6,8,9]: 570570
Query 3: splitting [6,7]: 40930
Query 3: splitting [6,7,9]: 158380
Query 3: splitting [6,7,8]: 204650
Query 3: splitting [6,7,8,9]: 613950
[info] *** Test still running after 3 minutes, 36 seconds: suite name: GeneralSkewSuite, test name: Query 3: 10-Table Join.
Query 3: splitting [5]: 38038
Query 3: splitting [5,9]: 146604
Query 3: splitting [5,8]: 190190
Query 3: splitting [5,8,9]: 570570
[info] - Query 3: 10-Table Join (4 minutes, 5 seconds)
Query 4: answer = 0
Query 4: splitting []: 0
Query 4: splitting [8]: 0
Query 4: splitting [5]: 0
Query 4: splitting [5,8]: 0
Query 4: splitting [4]: 0
Query 4: splitting [4,8]: 0
Query 4: splitting [4,5]: 0
Query 4: splitting [4,5,8]: 0
Query 4: random splits
Query 4: splitting []: 0
Query 4: splitting [9]: 45784
Query 4: splitting [8]: 0
Query 4: splitting [8,9]: 45784
Query 4: splitting [7]: 0
Query 4: splitting [7,9]: 51080
Query 4: splitting [7,8]: 0
Query 4: splitting [7,8,9]: 51080
Query 4: splitting [6]: 0
Query 4: splitting [6,9]: 45784
Query 4: splitting [6,8]: 0
Query 4: splitting [6,8,9]: 45784
Query 4: splitting [6,7]: 0
Query 4: splitting [6,7,9]: 51080
Query 4: splitting [6,7,8]: 0
Query 4: splitting [6,7,8,9]: 51080
Query 4: splitting [5]: 0
Query 4: splitting [5,9]: 45784
Query 4: splitting [5,8]: 0
Query 4: splitting [5,8,9]: 45784
[info] - Query 4: 10-Table Join (4 minutes, 2 seconds)
Query 5: answer = 364
Query 5: splitting []: 364
Query 5: splitting [2]: 364
Query 5: splitting [1]: 364
Query 5: splitting [1,2]: 364
Query 5: random splits
Query 5: splitting []: 364
Query 5: splitting [9]: 1092
Query 5: splitting [8]: 1820
Query 5: splitting [8,9]: 5460
Query 5: splitting [7]: 1820
Query 5: splitting [7,9]: 5460
Query 5: splitting [7,8]: 9100
Query 5: splitting [7,8,9]: 27300
Query 5: splitting [6]: 1233
Query 5: splitting [6,9]: 3699
Query 5: splitting [6,8]: 6165
Query 5: splitting [6,8,9]: 18495
Query 5: splitting [6,7]: 6165
Query 5: splitting [6,7,9]: 18495
Query 5: splitting [6,7,8]: 30825
Query 5: splitting [6,7,8,9]: 92475
Query 5: splitting [5]: 1083
Query 5: splitting [5,9]: 3249
Query 5: splitting [5,8]: 5415
[info] *** Test still running after 5 minutes, 28 seconds: suite name: GeneralSkewSuite, test name: Query 5: 10-Table Join with Agg.
Query 5: splitting [5,8,9]: 16245
[info] - Query 5: 10-Table Join with Agg (5 minutes, 50 seconds)
Query 6: answer = 898
Query 6: splitting []: 898
Query 6: splitting [8]: 898
Query 6: splitting [7]: 898
Query 6: splitting [7,8]: 898
Query 6: splitting [6]: 898
Query 6: splitting [6,8]: 898
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,8]: 898
Query 6: random splits
Query 6: splitting []: 898
Query 6: splitting [9]: 1116
Query 6: splitting [8]: 898
Query 6: splitting [8,9]: 1116
Query 6: splitting [7]: 898
Query 6: splitting [7,9]: 1116
Query 6: splitting [7,8]: 898
Query 6: splitting [7,8,9]: 1116
Query 6: splitting [6]: 898
Query 6: splitting [6,9]: 1116
Query 6: splitting [6,8]: 898
Query 6: splitting [6,8,9]: 1116
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,9]: 1116
[info] *** Test still running after 4 minutes, 38 seconds: suite name: GeneralSkewSuite, test name: Query 6: 10-Table Join with Agg.
Query 6: splitting [6,7,8]: 898
Query 6: splitting [6,7,8,9]: 1116
Query 6: splitting [5]: 2758
Query 6: splitting [5,9]: 3318
Query 6: splitting [5,8]: 2758
Query 6: splitting [5,8,9]: 3318
[info] - Query 6: 10-Table Join with Agg (7 minutes, 51 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #34108:
URL: https://github.com/apache/spark/pull/34108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927295366
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #34108:
URL: https://github.com/apache/spark/pull/34108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927256317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927291551
**[Test build #143634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143634/testReport)** for PR 34108 at commit [`07588a0`](https://github.com/apache/spark/commit/07588a06181667d101c91cedc0b6fa61f4c45100).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927242801
**[Test build #143634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143634/testReport)** for PR 34108 at commit [`07588a0`](https://github.com/apache/spark/commit/07588a06181667d101c91cedc0b6fa61f4c45100).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927256317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927295366
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927242801
**[Test build #143634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143634/testReport)** for PR 34108 at commit [`07588a0`](https://github.com/apache/spark/commit/07588a06181667d101c91cedc0b6fa61f4c45100).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927248010
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org