You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/22 06:42:23 UTC
[GitHub] [spark] ulysses-you opened a new pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
ulysses-you opened a new pull request #34069:
URL: https://github.com/apache/spark/pull/34069
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add a new hint `BROADCAST_NL`
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
For the join if one side is small and other side is large, the shuffle overhead is also very big. Due to the
bhj limitation, we can only broadcast right side for left join and left side for right join. So for the other case, we can try to use `BroadcastNestedLoopJoin` as the join strategy.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
yes, new hint
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Add new test in `JoinHintSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924862975
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143500/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926354328
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48095/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938574376
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926488069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143585/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926332383
**[Test build #143585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143585/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924704209
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r713766244
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala
##########
@@ -35,21 +35,27 @@ import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNes
* stage in case of the larger join child relation finishes before the smaller relation. Note
* that this rule needs to be applied before regular join strategies.
*/
-object LogicalQueryStageStrategy extends Strategy with PredicateHelper {
+object LogicalQueryStageStrategy extends Strategy with PredicateHelper with JoinSelectionHelper {
private def isBroadcastStage(plan: LogicalPlan): Boolean = plan match {
case LogicalQueryStage(_, _: BroadcastQueryStageExec) => true
case _ => false
}
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
+ case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint)
if isBroadcastStage(left) || isBroadcastStage(right) =>
val buildSide = if (isBroadcastStage(left)) BuildLeft else BuildRight
- Seq(BroadcastHashJoinExec(
Review comment:
We would never get `BroadcastNestedLoopJoinExec` if the join is `ExtractEquiJoinKeys ` before. And after this PR, the only way is hint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924832280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925490994
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925490994
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r713748928
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala
##########
@@ -35,21 +35,27 @@ import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNes
* stage in case of the larger join child relation finishes before the smaller relation. Note
* that this rule needs to be applied before regular join strategies.
*/
-object LogicalQueryStageStrategy extends Strategy with PredicateHelper {
+object LogicalQueryStageStrategy extends Strategy with PredicateHelper with JoinSelectionHelper {
private def isBroadcastStage(plan: LogicalPlan): Boolean = plan match {
case LogicalQueryStage(_, _: BroadcastQueryStageExec) => true
case _ => false
}
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
+ case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint)
if isBroadcastStage(left) || isBroadcastStage(right) =>
val buildSide = if (isBroadcastStage(left)) BuildLeft else BuildRight
- Seq(BroadcastHashJoinExec(
Review comment:
is this a long-standing bug? e.g. `BroadcastNestedLoopJoinExec` may become `BroadcastHashJoinExec` in AQE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r714933792
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala
##########
@@ -35,21 +35,27 @@ import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNes
* stage in case of the larger join child relation finishes before the smaller relation. Note
* that this rule needs to be applied before regular join strategies.
*/
-object LogicalQueryStageStrategy extends Strategy with PredicateHelper {
+object LogicalQueryStageStrategy extends Strategy with PredicateHelper with JoinSelectionHelper {
private def isBroadcastStage(plan: LogicalPlan): Boolean = plan match {
case LogicalQueryStage(_, _: BroadcastQueryStageExec) => true
case _ => false
}
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
+ case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint)
if isBroadcastStage(left) || isBroadcastStage(right) =>
val buildSide = if (isBroadcastStage(left)) BuildLeft else BuildRight
- Seq(BroadcastHashJoinExec(
- leftKeys, rightKeys, joinType, buildSide, otherCondition, planLater(left),
- planLater(right)))
+ if ((hintToBroadcastNLLeft(hint) && isBroadcastStage(left)) ||
+ hintToBroadcastNLRight(hint) && isBroadcastStage(right)) {
Review comment:
The code here looks a bit hacky to me, as it assumes how `BroadcastNestedLoopJoinExec` can be generated.
I feel a clearer way is to update `JoinSelection` and make it consider the broadcast stage directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-953133668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952968243
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938539655
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r713742152
##########
File path: docs/sql-ref-syntax-qry-select-hints.md
##########
@@ -126,6 +126,10 @@ Join hints allow users to suggest the join strategy that Spark should use. Prior
Suggests that Spark use shuffle-and-replicate nested loop join.
+* **BROADCAST_NL**
+
+ Suggests that Spark use broadcast nested loop join. The join side with the hint will be broadcast regardless of `autoBroadcastJoinThreshold`. If both sides of the join have the broadcast hints, the one with the smaller size (based on stats) will be broadcast.
Review comment:
should we highlight that this hint is useful when the join type is not supported by hash join? and list these join types, such as full outer join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924862975
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143500/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926377057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48095/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924798102
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-1030450621
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925553082
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143524/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924647102
**[Test build #143500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143500/testReport)** for PR 34069 at commit [`ed70065`](https://github.com/apache/spark/commit/ed7006560a56b6e4be2fde86d84aab289afe4606).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925488226
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938580781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924704234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952860664
**[Test build #144658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144658/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952860664
**[Test build #144658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144658/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r714417205
##########
File path: docs/sql-ref-syntax-qry-select-hints.md
##########
@@ -126,6 +126,10 @@ Join hints allow users to suggest the join strategy that Spark should use. Prior
Suggests that Spark use shuffle-and-replicate nested loop join.
+* **BROADCAST_NL**
+
+ Suggests that Spark use broadcast nested loop join. The join side with the hint will be broadcast regardless of `autoBroadcastJoinThreshold`. If both sides of the join have the broadcast hints, the one with the smaller size (based on stats) will be broadcast. Note that, this hint is only useful if the join type and build side is not supported by hash join. i.e. full outer join, left outer/semi/anti join with build left, right outer join with build right.
+
Review comment:
it should be broadcast hash join, let me make it clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924832280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925016245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r713766959
##########
File path: docs/sql-ref-syntax-qry-select-hints.md
##########
@@ -126,6 +126,10 @@ Join hints allow users to suggest the join strategy that Spark should use. Prior
Suggests that Spark use shuffle-and-replicate nested loop join.
+* **BROADCAST_NL**
+
+ Suggests that Spark use broadcast nested loop join. The join side with the hint will be broadcast regardless of `autoBroadcastJoinThreshold`. If both sides of the join have the broadcast hints, the one with the smaller size (based on stats) will be broadcast.
Review comment:
yea, updated the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925454881
**[Test build #143524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143524/testReport)** for PR 34069 at commit [`2aa2ee0`](https://github.com/apache/spark/commit/2aa2ee06a8110a9d2d7efe36aadc83afd75b71c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938705276
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144020/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #34069:
URL: https://github.com/apache/spark/pull/34069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952841591
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-953130688
**[Test build #144658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144658/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-953133668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-927248201
@ulysses-you - I see. With data skew, it makes sense to me, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926373023
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924647102
**[Test build #143500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143500/testReport)** for PR 34069 at commit [`ed70065`](https://github.com/apache/spark/commit/ed7006560a56b6e4be2fde86d84aab289afe4606).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926377045
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48095/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924776730
**[Test build #143505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143505/testReport)** for PR 34069 at commit [`d15eb78`](https://github.com/apache/spark/commit/d15eb784a80be4390f9bf87c0b0eee1cc5d7bd37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938698537
**[Test build #144020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144020/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925545273
**[Test build #143524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143524/testReport)** for PR 34069 at commit [`2aa2ee0`](https://github.com/apache/spark/commit/2aa2ee06a8110a9d2d7efe36aadc83afd75b71c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925470007
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926486210
**[Test build #143585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143585/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926332383
**[Test build #143585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143585/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925454881
**[Test build #143524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143524/testReport)** for PR 34069 at commit [`2aa2ee0`](https://github.com/apache/spark/commit/2aa2ee06a8110a9d2d7efe36aadc83afd75b71c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952946298
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924832249
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924776730
**[Test build #143505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143505/testReport)** for PR 34069 at commit [`d15eb78`](https://github.com/apache/spark/commit/d15eb784a80be4390f9bf87c0b0eee1cc5d7bd37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925016245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-927217863
hi @c21 , I agree. In general bnlj is much slower than smj. I find some extreme case that a left join with very small left side and large right side, and unfortunately the right side is also skewed. Then smj does not work good, even failed with OOM at skewed partition.
Here a simple benchmark with my local side:
```scala
spark.range(0, 10000000).selectExpr("id % 1 as c1", "id as c2").repartition(100).createOrReplaceTempView("t1")
spark.range(0, 10).selectExpr("id as c1").createOrReplaceTempView("t2")
// 5s
spark.sql("select /*+ merge(t2) */ count(*) from t2 left join t1 on t1.c1 = t2.c1").collect
// 3s
spark.sql("select /*+ broadcast_nl(t2) */ count(*) from t2 left join t1 on t1.c1 = t2.c1").collect
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924854584
**[Test build #143500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143500/testReport)** for PR 34069 at commit [`ed70065`](https://github.com/apache/spark/commit/ed7006560a56b6e4be2fde86d84aab289afe4606).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924633672
cc @cloud-fan @viirya @yaooqinn @maryannxue thank you for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938705276
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144020/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926377057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48095/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926372976
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924671655
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926373023
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-924704234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925007977
**[Test build #143505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143505/testReport)** for PR 34069 at commit [`d15eb78`](https://github.com/apache/spark/commit/d15eb784a80be4390f9bf87c0b0eee1cc5d7bd37).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952968243
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-952901481
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926356585
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r715274411
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala
##########
@@ -35,21 +35,27 @@ import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNes
* stage in case of the larger join child relation finishes before the smaller relation. Note
* that this rule needs to be applied before regular join strategies.
*/
-object LogicalQueryStageStrategy extends Strategy with PredicateHelper {
+object LogicalQueryStageStrategy extends Strategy with PredicateHelper with JoinSelectionHelper {
private def isBroadcastStage(plan: LogicalPlan): Boolean = plan match {
case LogicalQueryStage(_, _: BroadcastQueryStageExec) => true
case _ => false
}
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
+ case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint)
if isBroadcastStage(left) || isBroadcastStage(right) =>
val buildSide = if (isBroadcastStage(left)) BuildLeft else BuildRight
- Seq(BroadcastHashJoinExec(
- leftKeys, rightKeys, joinType, buildSide, otherCondition, planLater(left),
- planLater(right)))
+ if ((hintToBroadcastNLLeft(hint) && isBroadcastStage(left)) ||
+ hintToBroadcastNLRight(hint) && isBroadcastStage(right)) {
Review comment:
Not sure it will be clearer, the `LogicalQueryStageStrategy` is used to retain the origin broadcast join so it should know how broadcast join generated (include bhj and bnlj).
If we move this into `JoinSelection`, it looks like:
```scala
// LogicalQueryStageStrategy
case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint) =>
q.physicalPlan :: Nil
...
```
```scala
// JoinSelection
case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, otherCondition, _,
left, right, hint)
if isBroadcastStage(left) || isBroadcastStage(right) =>
val buildSide = if (isBroadcastStage(left)) BuildLeft else BuildRight
if ((hintToBroadcastNLLeft(hint) && isBroadcastStage(left)) ||
hintToBroadcastNLRight(hint) && isBroadcastStage(right)) {
Seq(BroadcastNestedLoopJoinExec(
planLater(left), planLater(right), buildSide, joinType, j.condition))
} else {
Seq(BroadcastHashJoinExec(
leftKeys, rightKeys, joinType, buildSide, otherCondition, planLater(left),
planLater(right)))
}
// the old equi-join code
case j @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, nonEquiCond,
_, left, right, hint) =>
...
```
Or, we can even move the whole code of `LogicalQueryStageStrategy` into `JoinSelection`. But for me, I perfer the current approach which makes broadcast stage easy to maintain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-926488069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143585/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938509858
**[Test build #144020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144020/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938580781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938509858
**[Test build #144020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144020/testReport)** for PR 34069 at commit [`4ed226f`](https://github.com/apache/spark/commit/4ed226f66643523f3661b53a28517383cf1f0eb5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34069:
URL: https://github.com/apache/spark/pull/34069#discussion_r714218856
##########
File path: docs/sql-ref-syntax-qry-select-hints.md
##########
@@ -126,6 +126,10 @@ Join hints allow users to suggest the join strategy that Spark should use. Prior
Suggests that Spark use shuffle-and-replicate nested loop join.
+* **BROADCAST_NL**
+
+ Suggests that Spark use broadcast nested loop join. The join side with the hint will be broadcast regardless of `autoBroadcastJoinThreshold`. If both sides of the join have the broadcast hints, the one with the smaller size (based on stats) will be broadcast. Note that, this hint is only useful if the join type and build side is not supported by hash join. i.e. full outer join, left outer/semi/anti join with build left, right outer join with build right.
+
Review comment:
`hash join` means broadcast or shuffle hash join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-925553082
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143524/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34069:
URL: https://github.com/apache/spark/pull/34069#issuecomment-938488242
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org