You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/04 11:24:52 UTC
[GitHub] [spark] wang-zhun opened a new pull request, #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
wang-zhun opened a new pull request, #37406:
URL: https://github.com/apache/spark/pull/37406
### What changes were proposed in this pull request?
Solve the data skew on the stream side in `BroadcastHashJoin`
- When data skew needs to introduce additional shuffle, support forcibly solve the data skew problem through `spark.sql.adaptive.forceOptimizeSkewedJoin`
- If data skew optimization is performed, `LocalShuffle` optimization will not be performed, otherwise the skew optimization will not take effect.
### Why are the changes needed?
In the actual production environment, data skew will slow down the task execution time
After solving the data skew
### How was this patch tested?
UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHzxp commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
CHzxp commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1206140696
you do not use AQE ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
URL: https://github.com/apache/spark/pull/37406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wang-zhun commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
wang-zhun commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1206294767
> you do not use AQE ?
Turning off AQE will be a `SortMergeJoin`, we need to turn on AQE and solve the data skew
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1325796418
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1205631569
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wang-zhun commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by "wang-zhun (via GitHub)" <gi...@apache.org>.
wang-zhun commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1644977407
> @wang-zhun I also found this problem. It seems that the tilt problem cannot be handled normally in the dynamic broadcast join. Does this pr community plan to merge? The existing method can only close the dynamic join
@thomasg19930417 At the moment, we haven't received feedback from the community. You can create a new pull request to address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] thomasg19930417 commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1644869323
@wang-zhun I also found this problem. It seems that the tilt problem cannot be handled normally in the dynamic broadcast join. Does this pr community plan to merge? The existing method can only close the dynamic join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHzxp commented on pull request #37406: [SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin
Posted by GitBox <gi...@apache.org>.
CHzxp commented on PR #37406:
URL: https://github.com/apache/spark/pull/37406#issuecomment-1214855684
so, i guess turning off AQE is aim to force execute broadcastHashJoin? because AQE will change plan to SortMergeJoin according to stage statistics, and AQE will solve the stream side skew by add external shuffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org