You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/08 09:46:39 UTC
[GitHub] [spark] leanken opened a new pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
leanken opened a new pull request #29389:
URL: https://github.com/apache/spark/pull/29389
### What changes were proposed in this pull request?
In [SPARK-32290](https://issues.apache.org/jira/browse/SPARK-32290), we introduced several new types of HashedRelation.
* EmptyHashedRelation
* EmptyHashedRelationWithAllNullKeys
They were all limited to used only in NAAJ scenario. But as for a improvement, EmptyHashedRelation could also be used in Normal AntiJoin for fast stop, and as for in AQE, we can even eliminate anti join when we knew that buildSide is empty.
This Patch including two changes.
* In Non-AQE, using EmptyHashedRelation to do fast stop for common anti join as well
* In AQE, eliminate anti join if buildSide is a EmptyHashedRelation of ShuffleWriteRecord is 0
### Why are the changes needed?
LeftAntiJoin could apply `fast stop` when BuildSide is Empty, While within AQE, we can even eliminate the anti join. Th
is should be a performance improvement in AntiJoin.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
* added case in AdaptiveQueryExecSuite.
* added case in HashedRelationSuite.
* Make sure SubquerySuite JoinSuite SQLQueryTestSuite passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467690876
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##########
@@ -82,6 +82,8 @@ case class BroadcastExchangeExec(
"buildTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to build"),
"broadcastTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to broadcast"))
+ private var knownRowCount: Option[BigInt] = None
Review comment:
Sure will do the update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671197153
@cloud-fan updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671433077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671644656
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670991633
**[Test build #127231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127231/testReport)** for PR 29389 at commit [`25dd444`](https://github.com/apache/spark/commit/25dd44417e118ea87470c00ed11393055eb2ee52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671718124
@cloud-fan Test passed, ready to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670991720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670996570
**[Test build #127233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127233/testReport)** for PR 29389 at commit [`2063288`](https://github.com/apache/spark/commit/2063288e072a266218b57772ada874d35432b344).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671432322
**[Test build #127286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127286/testReport)** for PR 29389 at commit [`c8eae30`](https://github.com/apache/spark/commit/c8eae30493ba7df0c5918a00d0b1c58c681b27ca).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671433077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671587850
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467691348
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -950,8 +950,18 @@ trait NullAwareHashedRelation extends HashedRelation with Externalizable {
/**
* A special HashedRelation indicates it built from a empty input:Iterator[InternalRow].
+ * get & getValue will return null just like
Review comment:
yes. It will be called. EmptyHashedRelation now is also applied at all joinType. I confirmed it while running UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467945765
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
Review comment:
I was also consider doing it inside BHJ because in ShuffledHashJoinExec
the hashedRelation is built among generated-code, it might be hard to just return hashedRelation in prepareRelation call.
```
protected override def prepareRelation(ctx: CodegenContext): (String, Boolean) = {
val thisPlan = ctx.addReferenceObj("plan", this)
val clsName = classOf[HashedRelation].getName
// Inline mutable state since not many join operations in a task
val relationTerm = ctx.addMutableState(clsName, "relation",
v => s"$v = $thisPlan.buildHashedRelation(inputs[1]);", forceInline = true)
(relationTerm, false)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671433050
@cloud-fan updated. refine prepareRelation to return
```
private[joins] case class HashedRelationInfo(
relationTerm: String,
keyIsUnique: Boolean,
isEmptyHashedRelation: Boolean)
```
Code change int BHJ reverted, and do fast stop in HashJoin.scala codegenAnti using isEmptyHashedRelation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467683352
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -950,8 +950,18 @@ trait NullAwareHashedRelation extends HashedRelation with Externalizable {
/**
* A special HashedRelation indicates it built from a empty input:Iterator[InternalRow].
+ * get & getValue will return null just like
+ * empty LongHashedRelation or empty UnsafeHashedRelation does.
*/
object EmptyHashedRelation extends NullAwareHashedRelation {
+ override def get(key: Long): Iterator[InternalRow] = null
Review comment:
are you sure `UnsafeHashedRelation` returns null not `Iterator.empty`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670996570
**[Test build #127233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127233/testReport)** for PR 29389 at commit [`2063288`](https://github.com/apache/spark/commit/2063288e072a266218b57772ada874d35432b344).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671009262
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671015466
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671009262
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671196620
**[Test build #127249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127249/testReport)** for PR 29389 at commit [`257745a`](https://github.com/apache/spark/commit/257745ae314de2ea7a186916aa7b47ce66a60ca9).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671646391
**[Test build #127296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127296/testReport)** for PR 29389 at commit [`5309b66`](https://github.com/apache/spark/commit/5309b66b23af2ffe7999edbd8f0c11f26628805d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671318532
**[Test build #127270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127270/testReport)** for PR 29389 at commit [`381db11`](https://github.com/apache/spark/commit/381db1115c1543513c77369306ca11a3ce00ad47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671015287
**[Test build #127233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127233/testReport)** for PR 29389 at commit [`2063288`](https://github.com/apache/spark/commit/2063288e072a266218b57772ada874d35432b344).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670852395
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670991720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671718149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29389:
URL: https://github.com/apache/spark/pull/29389
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671587850
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467691030
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -950,8 +950,18 @@ trait NullAwareHashedRelation extends HashedRelation with Externalizable {
/**
* A special HashedRelation indicates it built from a empty input:Iterator[InternalRow].
+ * get & getValue will return null just like
+ * empty LongHashedRelation or empty UnsafeHashedRelation does.
*/
object EmptyHashedRelation extends NullAwareHashedRelation {
+ override def get(key: Long): Iterator[InternalRow] = null
Review comment:
```
override def get(key: InternalRow): Iterator[InternalRow] = {
val unsafeKey = key.asInstanceOf[UnsafeRow]
val map = binaryMap // avoid the compiler error
val loc = new map.Location // this could be allocated in stack
binaryMap.safeLookup(unsafeKey.getBaseObject, unsafeKey.getBaseOffset,
unsafeKey.getSizeInBytes, loc, unsafeKey.hashCode())
if (loc.isDefined) {
new Iterator[UnsafeRow] {
private var _hasNext = true
override def hasNext: Boolean = _hasNext
override def next(): UnsafeRow = {
resultRow.pointTo(loc.getValueBase, loc.getValueOffset, loc.getValueLength)
_hasNext = loc.nextValue()
resultRow
}
}
} else {
null // return null if not match
}
}
def getValue(key: InternalRow): InternalRow = {
val unsafeKey = key.asInstanceOf[UnsafeRow]
val map = binaryMap // avoid the compiler error
val loc = new map.Location // this could be allocated in stack
binaryMap.safeLookup(unsafeKey.getBaseObject, unsafeKey.getBaseOffset,
unsafeKey.getSizeInBytes, loc, unsafeKey.hashCode())
if (loc.isDefined) {
resultRow.pointTo(loc.getValueBase, loc.getValueOffset, loc.getValueLength)
resultRow
} else {
null // return null if not match
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467525605
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -125,7 +125,7 @@ private[execution] object HashedRelation {
* [number of keys]
* [size of key] [size of value] [key bytes] [bytes for value]
*/
-private[joins] class UnsafeHashedRelation(
+class UnsafeHashedRelation(
Review comment:
Ah. this is historical code. A question, should we consider the same when
LongHashedRelation.keys.isEmpty
UnsafeHashedRelation.keys.isEmpty
equalTo EmptyHashedRelation
if it does, we need this private[joins] to be removed. I am not quite sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467523040
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##########
@@ -82,6 +82,8 @@ case class BroadcastExchangeExec(
"buildTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to build"),
"broadcastTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to broadcast"))
+ var zeroNumRows: Boolean = false
Review comment:
nit: How about rewriting it like this?
```
+ private var knownRowCount: Option[BigInt] = None
+
override def runtimeStatistics: Statistics = {
val dataSize = metrics("dataSize").value
- Statistics(dataSize, rowCount = if (zeroNumRows) Some(0L) else None)
+ Statistics(dataSize, rowCount = knownRowCount)
}
@transient
@@ -128,6 +128,9 @@ case class BroadcastExchangeExec(
val relation = mode.transform(input, Some(numRows))
val dataSize = relation match {
+ case EmptyHashedRelation =>
+ knownRowCount = Some(0L)
+ 0L
case map: HashedRelation =>
map.estimatedSize
case arr: Array[InternalRow] =>
@@ -137,7 +140,6 @@ case class BroadcastExchangeExec(
s"type: ${relation.getClass.getName}")
}
- zeroNumRows = relation == EmptyHashedRelation
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671718149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670852306
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671197680
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467527399
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -125,7 +125,7 @@ private[execution] object HashedRelation {
* [number of keys]
* [size of key] [size of value] [key bytes] [bytes for value]
*/
-private[joins] class UnsafeHashedRelation(
+class UnsafeHashedRelation(
Review comment:
Ah, I see. it is a trivial change and it looks okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670852395
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670852306
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670991526
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671220770
Update RP desc and Jira desc accordingly, @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671015466
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671199044
**[Test build #127270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127270/testReport)** for PR 29389 at commit [`381db11`](https://github.com/apache/spark/commit/381db1115c1543513c77369306ca11a3ce00ad47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671717620
**[Test build #127296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127296/testReport)** for PR 29389 at commit [`5309b66`](https://github.com/apache/spark/commit/5309b66b23af2ffe7999edbd8f0c11f26628805d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671130187
**[Test build #127249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127249/testReport)** for PR 29389 at commit [`257745a`](https://github.com/apache/spark/commit/257745ae314de2ea7a186916aa7b47ce66a60ca9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r468039532
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala
##########
@@ -81,13 +81,13 @@ case class ShuffledHashJoinExec(
override def needCopyResult: Boolean = true
- protected override def prepareRelation(ctx: CodegenContext): (String, Boolean) = {
+ protected override def prepareRelation(ctx: CodegenContext): HashedRelationInfo = {
val thisPlan = ctx.addReferenceObj("plan", this)
val clsName = classOf[HashedRelation].getName
// Inline mutable state since not many join operations in a task
val relationTerm = ctx.addMutableState(clsName, "relation",
v => s"$v = $thisPlan.buildHashedRelation(inputs[1]);", forceInline = true)
- (relationTerm, false)
+ HashedRelationInfo(relationTerm, false, false)
Review comment:
nit: `...(..., isKeyUnique = false, isEmpty = false)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671319387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671432322
**[Test build #127286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127286/testReport)** for PR 29389 at commit [`c8eae30`](https://github.com/apache/spark/commit/c8eae30493ba7df0c5918a00d0b1c58c681b27ca).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671319387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467522246
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateAntiJoin.scala
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * This optimization rule detects and eliminate a LeftAntiJoin when buildSide is empty.
+ */
+case class EliminateAntiJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ private def shouldEliminate(plan: LogicalPlan): Boolean = plan match {
Review comment:
nit: `should` -> `can`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671587856
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127286/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671199044
**[Test build #127270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127270/testReport)** for PR 29389 at commit [`381db11`](https://github.com/apache/spark/commit/381db1115c1543513c77369306ca11a3ce00ad47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671197686
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127249/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671197644
1. keep EmptyHashedRelation for common anti join fast stop both for NON-AQE and AQE
2. rewrite NAAJ to Empty LocalRelation when BuildSide is EmptyHashedRelationWithAllNullKeys with AQE on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670997405
You, too, @leanken ;) thanks for the work, anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467925917
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
Review comment:
After a second look, I think this is incorrect. We may go to the `else` branch, which calls `super.codegenAnti` and call `prepareBroadcast` again.
I think we should keep this method untouched, and update `HashJoin.codegenAnti` to fast stop with `EmptyHashedRelation`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670852066
@cloud-fan could you please have a look at this followup?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467646238
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateAntiJoin.scala
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * This optimization rule detects and eliminate a LeftAntiJoin when buildSide is empty.
+ */
+case class EliminateAntiJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ private def canEliminate(plan: LogicalPlan): Boolean = plan match {
+ case LogicalQueryStage(_, stage: QueryStageExec) if stage.resultOption.get().isDefined
+ && stage.getRuntimeStatistics.rowCount.isDefined
+ && stage.getRuntimeStatistics.rowCount.get == 0L => true
+ case _ => false
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDown {
+ // If the right side is empty, LeftAntiJoin simply returns the left side.
+ // Eliminate Join with left LogicalPlan instead.
+ case Join(left, right, LeftAnti, _, _)
+ if canEliminate(right) =>
Review comment:
nit. Shall we make this one line, `case Join(left, right, LeftAnti, _, _) if canEliminate(right) =>`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671220511
Update PR title => Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670996658
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467696510
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateAntiJoin.scala
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * This optimization rule detects and eliminate a LeftAntiJoin when buildSide is empty.
+ */
+case class EliminateAntiJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ private def canEliminate(plan: LogicalPlan): Boolean = plan match {
+ case LogicalQueryStage(_, stage: QueryStageExec) if stage.resultOption.get().isDefined
+ && stage.getRuntimeStatistics.rowCount.isDefined
+ && stage.getRuntimeStatistics.rowCount.get == 0L => true
+ case _ => false
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDown {
+ // If the right side is empty, LeftAntiJoin simply returns the left side.
+ // Eliminate Join with left LogicalPlan instead.
+ case Join(left, right, LeftAnti, _, _) if canEliminate(right) =>
Review comment:
This is not very useful, as the anti join is already very fast with the empty right side.
https://github.com/apache/spark/pull/29389/files#r467683621 is more useful, as for now we still need to consume all the left side data and return nothing. It will be a big improvement if we can directly convert the join node to an empty relation with AQE.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467697219
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateAntiJoin.scala
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * This optimization rule detects and eliminate a LeftAntiJoin when buildSide is empty.
+ */
+case class EliminateAntiJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ private def canEliminate(plan: LogicalPlan): Boolean = plan match {
+ case LogicalQueryStage(_, stage: QueryStageExec) if stage.resultOption.get().isDefined
+ && stage.getRuntimeStatistics.rowCount.isDefined
+ && stage.getRuntimeStatistics.rowCount.get == 0L => true
+ case _ => false
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDown {
+ // If the right side is empty, LeftAntiJoin simply returns the left side.
+ // Eliminate Join with left LogicalPlan instead.
+ case Join(left, right, LeftAnti, _, _) if canEliminate(right) =>
Review comment:
> This is not very useful, as the anti join is already very fast with the empty right side.
>
> https://github.com/apache/spark/pull/29389/files#r467683621 is more useful, as for now we still need to consume all the left side data and return nothing. It will be a big improvement if we can directly convert the join node to an empty relation with AQE.
In fact, we need to consume all the left side data and return all the rows when it's an EmptyHashedRelation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467682881
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##########
@@ -126,6 +128,9 @@ case class BroadcastExchangeExec(
val relation = mode.transform(input, Some(numRows))
val dataSize = relation match {
+ case EmptyHashedRelation =>
+ knownRowCount = Some(0L)
Review comment:
See L118. We already know the row count and we even use it to check `numRows >= MAX_BROADCAST_TABLE_ROWS`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r468038934
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
##########
@@ -29,6 +29,16 @@ import org.apache.spark.sql.execution.{CodegenSupport, ExplainUtils, RowIterator
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.types.{BooleanType, IntegralType, LongType}
+/**
+ * @param relationTerm variable name for HashedRelation
+ * @param keyIsUnique indicate whether keys of HashedRelation known to be unique in code-gen time
+ * @param isEmptyHashedRelation indicate whether it known to be EmptyHashedRelation in code-gen time
+ */
+private[joins] case class HashedRelationInfo(
+ relationTerm: String,
+ keyIsUnique: Boolean,
+ isEmptyHashedRelation: Boolean)
Review comment:
we can probably make the name shorter, just use `isEmpty`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467697807
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateAntiJoin.scala
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * This optimization rule detects and eliminate a LeftAntiJoin when buildSide is empty.
+ */
+case class EliminateAntiJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ private def canEliminate(plan: LogicalPlan): Boolean = plan match {
+ case LogicalQueryStage(_, stage: QueryStageExec) if stage.resultOption.get().isDefined
+ && stage.getRuntimeStatistics.rowCount.isDefined
+ && stage.getRuntimeStatistics.rowCount.get == 0L => true
+ case _ => false
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDown {
+ // If the right side is empty, LeftAntiJoin simply returns the left side.
+ // Eliminate Join with left LogicalPlan instead.
+ case Join(left, right, LeftAnti, _, _) if canEliminate(right) =>
Review comment:
> In fact, we need to consume all the left side data and return all the rows when it's an EmptyHashedRelation.
Yes, but this rule doesn't change it. By returning the left side plan, we still need to consume all the data of the left side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467683517
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -950,8 +950,18 @@ trait NullAwareHashedRelation extends HashedRelation with Externalizable {
/**
* A special HashedRelation indicates it built from a empty input:Iterator[InternalRow].
+ * get & getValue will return null just like
Review comment:
BTW where do we call these methods for `EmptyHashedRelation`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671646391
**[Test build #127296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127296/testReport)** for PR 29389 at commit [`5309b66`](https://github.com/apache/spark/commit/5309b66b23af2ffe7999edbd8f0c11f26628805d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671009126
**[Test build #127231 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127231/testReport)** for PR 29389 at commit [`25dd444`](https://github.com/apache/spark/commit/25dd44417e118ea87470c00ed11393055eb2ee52).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671644656
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467646539
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val numOutput = metricTerm(ctx, "numOutputRows")
+ if (broadcastRelation.value == EmptyHashedRelation) {
+ s"""
+ |// If the right side is empty, AntiJoin simply returns the left side.
+ |$numOutput.add(1);
+ |${consume(ctx, input)}
+ """.stripMargin
Review comment:
indentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670996334
@maropu updated. thanks for doing this in weekend. ^_^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671199454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467683621
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val numOutput = metricTerm(ctx, "numOutputRows")
+ if (broadcastRelation.value == EmptyHashedRelation) {
+ s"""
+ |// If the right side is empty, AntiJoin simply returns the left side.
+ |$numOutput.add(1);
+ |${consume(ctx, input)}
+ """.stripMargin
+ } else if (isNullAwareAntiJoin) {
val (keyEv, anyNull) = genStreamSideJoinKey(ctx, input)
val (matched, _, _) = getJoinCondition(ctx, input)
- val numOutput = metricTerm(ctx, "numOutputRows")
- if (broadcastRelation.value == EmptyHashedRelation) {
- s"""
- |// If the right side is empty, NAAJ simply returns the left side.
- |$numOutput.add(1);
- |${consume(ctx, input)}
- """.stripMargin
- } else if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
+ if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
Review comment:
shall we handle this case in AQE as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467942977
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
Review comment:
I thought about this problem too.
HashJoin.scala does not have the API prepareBroadcast, only the following prepareRelation available, which will not return HashedRelation.
```
protected def prepareRelation(ctx: CodegenContext): (String, Boolean)
```
I was thinking that call prepareBroadcast twice is OK since broadcast.Broadcast[HashedRelation] is a future call, and the collect and build action will only happen once, but the relationTerm will be generated twice
is it ok to just change prepareRelation to return a (String, Boolean, HashedRelation) ??
tell me when you decided which way to go. @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671130187
**[Test build #127249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127249/testReport)** for PR 29389 at commit [`257745a`](https://github.com/apache/spark/commit/257745ae314de2ea7a186916aa7b47ce66a60ca9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467691971
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val numOutput = metricTerm(ctx, "numOutputRows")
+ if (broadcastRelation.value == EmptyHashedRelation) {
+ s"""
+ |// If the right side is empty, AntiJoin simply returns the left side.
+ |$numOutput.add(1);
+ |${consume(ctx, input)}
+ """.stripMargin
+ } else if (isNullAwareAntiJoin) {
val (keyEv, anyNull) = genStreamSideJoinKey(ctx, input)
val (matched, _, _) = getJoinCondition(ctx, input)
- val numOutput = metricTerm(ctx, "numOutputRows")
- if (broadcastRelation.value == EmptyHashedRelation) {
- s"""
- |// If the right side is empty, NAAJ simply returns the left side.
- |$numOutput.add(1);
- |${consume(ctx, input)}
- """.stripMargin
- } else if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
+ if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
Review comment:
I think EmptyHashedRelationWithAllNullKeys should only be handled in NAAJ scenario. It might be needed to consider in my next follow up => support NAAJ in ShuffleHashJoinExec.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670991633
**[Test build #127231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127231/testReport)** for PR 29389 at commit [`25dd444`](https://github.com/apache/spark/commit/25dd44417e118ea87470c00ed11393055eb2ee52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467696488
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -223,19 +223,19 @@ case class BroadcastHashJoinExec(
* Handles NULL-aware anti join (NAAJ) separately here.
*/
protected override def codegenAnti(ctx: CodegenContext, input: Seq[ExprCode]): String = {
- if (isNullAwareAntiJoin) {
- val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val (broadcastRelation, relationTerm) = prepareBroadcast(ctx)
+ val numOutput = metricTerm(ctx, "numOutputRows")
+ if (broadcastRelation.value == EmptyHashedRelation) {
+ s"""
+ |// If the right side is empty, AntiJoin simply returns the left side.
+ |$numOutput.add(1);
+ |${consume(ctx, input)}
+ """.stripMargin
+ } else if (isNullAwareAntiJoin) {
val (keyEv, anyNull) = genStreamSideJoinKey(ctx, input)
val (matched, _, _) = getJoinCondition(ctx, input)
- val numOutput = metricTerm(ctx, "numOutputRows")
- if (broadcastRelation.value == EmptyHashedRelation) {
- s"""
- |// If the right side is empty, NAAJ simply returns the left side.
- |$numOutput.add(1);
- |${consume(ctx, input)}
- """.stripMargin
- } else if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
+ if (broadcastRelation.value == EmptyHashedRelationWithAllNullKeys) {
Review comment:
do you mean to eliminate NAAJ BroadcastJoin if it is a EmptyHashedRelationWithAllNullKeys?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671130359
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671197680
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671587006
**[Test build #127286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127286/testReport)** for PR 29389 at commit [`c8eae30`](https://github.com/apache/spark/commit/c8eae30493ba7df0c5918a00d0b1c58c681b27ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-670996658
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671213173
@leanken please also update the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671130359
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467682537
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##########
@@ -82,6 +82,8 @@ case class BroadcastExchangeExec(
"buildTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to build"),
"broadcastTime" -> SQLMetrics.createTimingMetric(sparkContext, "time to broadcast"))
+ private var knownRowCount: Option[BigInt] = None
Review comment:
can we be consistent and use `metrics` to hold rowCount? You can follow how we track `dataSize`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29389: [SPARK-32573][SQL] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671753459
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671199454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467646746
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -950,8 +950,18 @@ trait NullAwareHashedRelation extends HashedRelation with Externalizable {
/**
* A special HashedRelation indicates it built from a empty input:Iterator[InternalRow].
+ * get & getValue will return null just like
+ * Empty LongHashedRelation or Empty UnsafeHashedRelation does.
Review comment:
`Empty` -> `empty`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29389:
URL: https://github.com/apache/spark/pull/29389#discussion_r467523167
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
##########
@@ -125,7 +125,7 @@ private[execution] object HashedRelation {
* [number of keys]
* [size of key] [size of value] [key bytes] [bytes for value]
*/
-private[joins] class UnsafeHashedRelation(
+class UnsafeHashedRelation(
Review comment:
We need this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leanken commented on pull request #29389: [SPARK-32573][SQL] Eliminate Anti Join when BuildSide is Empty
Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #29389:
URL: https://github.com/apache/spark/pull/29389#issuecomment-671130039
@dongjoon-hyun updated, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org