You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/26 06:26:32 UTC
[GitHub] [spark] zhengruifeng commented on pull request #34108: [SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness
zhengruifeng commented on pull request #34108:
URL: https://github.com/apache/spark/pull/34108#issuecomment-927242653
this local result should be like:
```
Query 1: answer = 37714
Query 1: splitting []: 37714
Query 1: splitting [7]: 37714
Query 1: splitting [6]: 37714
Query 1: splitting [6,7]: 37714
Query 1: splitting [3]: 37714
Query 1: splitting [3,7]: 37714
Query 1: splitting [3,6]: 37714
Query 1: splitting [3,6,7]: 37714
Query 1: splitting [0]: 37714
Query 1: splitting [0,7]: 37714
Query 1: splitting [0,6]: 37714
Query 1: splitting [0,6,7]: 37714
Query 1: splitting [0,3]: 37714
Query 1: splitting [0,3,7]: 37714
Query 1: splitting [0,3,6]: 37714
Query 1: splitting [0,3,6,7]: 37714
Query 1: random splits
Query 1: splitting []: 37714
Query 1: splitting [9]: 113142
Query 1: splitting [8]: 155044
Query 1: splitting [8,9]: 465132
Query 1: splitting [7]: 37714
Query 1: splitting [7,9]: 113142
Query 1: splitting [7,8]: 155044
Query 1: splitting [7,8,9]: 465132
Query 1: splitting [6]: 37714
Query 1: splitting [6,9]: 113142
Query 1: splitting [6,8]: 155044
Query 1: splitting [6,8,9]: 465132
Query 1: splitting [6,7]: 37714
[info] *** Test still running after 4 minutes, 58 seconds: suite name: GeneralSkewSuite, test name: Query 1: 10-Table Join.
Query 1: splitting [6,7,9]: 113142
Query 1: splitting [6,7,8]: 155044
Query 1: splitting [6,7,8,9]: 465132
Query 1: splitting [5]: 37714
Query 1: splitting [5,9]: 113142
Query 1: splitting [5,8]: 155044
Query 1: splitting [5,8,9]: 465132
[info] - Query 1: 10-Table Join (7 minutes, 25 seconds)
Query 2: answer = 324
Query 2: splitting []: 324
Query 2: splitting [6]: 324
Query 2: splitting [2]: 324
Query 2: splitting [2,6]: 324
Query 2: splitting [1]: 324
Query 2: splitting [1,6]: 324
Query 2: splitting [1,2]: 324
Query 2: splitting [1,2,6]: 324
Query 2: random splits
Query 2: splitting []: 324
Query 2: splitting [9]: 14266
Query 2: splitting [8]: 35146
Query 2: splitting [8,9]: 151222
Query 2: splitting [7]: 540
Query 2: splitting [7,9]: 17110
Query 2: splitting [7,8]: 42870
Query 2: splitting [7,8,9]: 179690
Query 2: splitting [6]: 324
Query 2: splitting [6,9]: 14266
Query 2: splitting [6,8]: 35146
Query 2: splitting [6,8,9]: 151222
Query 2: splitting [6,7]: 540
Query 2: splitting [6,7,9]: 17110
Query 2: splitting [6,7,8]: 42870
[info] *** Test still running after 2 minutes, 33 seconds: suite name: GeneralSkewSuite, test name: Query 2: 10-Table Join.
Query 2: splitting [6,7,8,9]: 179690
Query 2: splitting [5]: 540
Query 2: splitting [5,9]: 26895
Query 2: splitting [5,8]: 56775
Query 2: splitting [5,8,9]: 256425
[info] - Query 2: 10-Table Join (3 minutes, 56 seconds)
Query 3: answer = 38038
Query 3: splitting []: 38038
Query 3: splitting [5]: 38038
Query 3: splitting [4]: 38038
Query 3: splitting [4,5]: 38038
Query 3: splitting [2]: 38038
Query 3: splitting [2,5]: 38038
Query 3: splitting [2,4]: 38038
Query 3: splitting [2,4,5]: 38038
Query 3: random splits
Query 3: splitting []: 38038
Query 3: splitting [9]: 146604
Query 3: splitting [8]: 190190
Query 3: splitting [8,9]: 570570
Query 3: splitting [7]: 40930
Query 3: splitting [7,9]: 158380
Query 3: splitting [7,8]: 204650
Query 3: splitting [7,8,9]: 613950
Query 3: splitting [6]: 38038
Query 3: splitting [6,9]: 146604
Query 3: splitting [6,8]: 190190
Query 3: splitting [6,8,9]: 570570
Query 3: splitting [6,7]: 40930
Query 3: splitting [6,7,9]: 158380
Query 3: splitting [6,7,8]: 204650
Query 3: splitting [6,7,8,9]: 613950
[info] *** Test still running after 3 minutes, 36 seconds: suite name: GeneralSkewSuite, test name: Query 3: 10-Table Join.
Query 3: splitting [5]: 38038
Query 3: splitting [5,9]: 146604
Query 3: splitting [5,8]: 190190
Query 3: splitting [5,8,9]: 570570
[info] - Query 3: 10-Table Join (4 minutes, 5 seconds)
Query 4: answer = 0
Query 4: splitting []: 0
Query 4: splitting [8]: 0
Query 4: splitting [5]: 0
Query 4: splitting [5,8]: 0
Query 4: splitting [4]: 0
Query 4: splitting [4,8]: 0
Query 4: splitting [4,5]: 0
Query 4: splitting [4,5,8]: 0
Query 4: random splits
Query 4: splitting []: 0
Query 4: splitting [9]: 45784
Query 4: splitting [8]: 0
Query 4: splitting [8,9]: 45784
Query 4: splitting [7]: 0
Query 4: splitting [7,9]: 51080
Query 4: splitting [7,8]: 0
Query 4: splitting [7,8,9]: 51080
Query 4: splitting [6]: 0
Query 4: splitting [6,9]: 45784
Query 4: splitting [6,8]: 0
Query 4: splitting [6,8,9]: 45784
Query 4: splitting [6,7]: 0
Query 4: splitting [6,7,9]: 51080
Query 4: splitting [6,7,8]: 0
Query 4: splitting [6,7,8,9]: 51080
Query 4: splitting [5]: 0
Query 4: splitting [5,9]: 45784
Query 4: splitting [5,8]: 0
Query 4: splitting [5,8,9]: 45784
[info] - Query 4: 10-Table Join (4 minutes, 2 seconds)
Query 5: answer = 364
Query 5: splitting []: 364
Query 5: splitting [2]: 364
Query 5: splitting [1]: 364
Query 5: splitting [1,2]: 364
Query 5: random splits
Query 5: splitting []: 364
Query 5: splitting [9]: 1092
Query 5: splitting [8]: 1820
Query 5: splitting [8,9]: 5460
Query 5: splitting [7]: 1820
Query 5: splitting [7,9]: 5460
Query 5: splitting [7,8]: 9100
Query 5: splitting [7,8,9]: 27300
Query 5: splitting [6]: 1233
Query 5: splitting [6,9]: 3699
Query 5: splitting [6,8]: 6165
Query 5: splitting [6,8,9]: 18495
Query 5: splitting [6,7]: 6165
Query 5: splitting [6,7,9]: 18495
Query 5: splitting [6,7,8]: 30825
Query 5: splitting [6,7,8,9]: 92475
Query 5: splitting [5]: 1083
Query 5: splitting [5,9]: 3249
Query 5: splitting [5,8]: 5415
[info] *** Test still running after 5 minutes, 28 seconds: suite name: GeneralSkewSuite, test name: Query 5: 10-Table Join with Agg.
Query 5: splitting [5,8,9]: 16245
[info] - Query 5: 10-Table Join with Agg (5 minutes, 50 seconds)
Query 6: answer = 898
Query 6: splitting []: 898
Query 6: splitting [8]: 898
Query 6: splitting [7]: 898
Query 6: splitting [7,8]: 898
Query 6: splitting [6]: 898
Query 6: splitting [6,8]: 898
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,8]: 898
Query 6: random splits
Query 6: splitting []: 898
Query 6: splitting [9]: 1116
Query 6: splitting [8]: 898
Query 6: splitting [8,9]: 1116
Query 6: splitting [7]: 898
Query 6: splitting [7,9]: 1116
Query 6: splitting [7,8]: 898
Query 6: splitting [7,8,9]: 1116
Query 6: splitting [6]: 898
Query 6: splitting [6,9]: 1116
Query 6: splitting [6,8]: 898
Query 6: splitting [6,8,9]: 1116
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,9]: 1116
[info] *** Test still running after 4 minutes, 38 seconds: suite name: GeneralSkewSuite, test name: Query 6: 10-Table Join with Agg.
Query 6: splitting [6,7,8]: 898
Query 6: splitting [6,7,8,9]: 1116
Query 6: splitting [5]: 2758
Query 6: splitting [5,9]: 3318
Query 6: splitting [5,8]: 2758
Query 6: splitting [5,8,9]: 3318
[info] - Query 6: 10-Table Join with Agg (7 minutes, 51 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org