You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/28 20:16:00 UTC

[GitHub] [spark] peter-toth opened a new pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

peter-toth opened a new pull request #29572:
URL: https://github.com/apache/spark/pull/29572


   ### What changes were proposed in this pull request?
   
   LeftSemi SortMergeJoin should not buffer all matching right side rows when bound condition is empty, this is unnecessary and can lead to performance degradation especially when spilling happens.
   
   ### Why are the changes needed?
   
   Performance improvement.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Existing UTs and TPCDS benchmarks.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482730876



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -662,7 +668,8 @@ private[joins] class SortMergeJoinScanner(
     bufferedIter: RowIterator,
     inMemoryThreshold: Int,
     spillThreshold: Int,
-    eagerCleanupResources: () => Unit) {
+    eagerCleanupResources: () => Unit,
+    bufferFirstOnly: Boolean) {

Review comment:
       nit: `bufferFirstOnly` -> `matchedBufferFirstOnly`? And, please add `@param`, too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482756735



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,9 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
-    new ExternalAppendOnlyUnsafeRowArray(inMemoryThreshold, spillThreshold)
+  private[this] val bufferedMatches: ExternalAppendOnlyUnsafeRowArray =
+    new ExternalAppendOnlyUnsafeRowArray(if (bufferFirstOnly) 1 else inMemoryThreshold,

Review comment:
       Not this change does it, but the one in `bufferMatchingRows()`. This change just avoids creating a buffer larger than 1.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686319796






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486756148



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686323730






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686304845


   **[Test build #128238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128238/testReport)** for PR 29572 at commit [`3937e4c`](https://github.com/apache/spark/commit/3937e4ca0b8631d785543c53d492eaa5b17b1ab4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686180452






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686547055


   Thanks for the review @cloud-fan, @maropu, @viirya. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486833087



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686319217


   **[Test build #128242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128242/testReport)** for PR 29572 at commit [`5cf3ab3`](https://github.com/apache/spark/commit/5cf3ab329d77b4cc525b2f4ca1992b75782a7f05).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686319796






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683129162


   **[Test build #128000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128000/testReport)** for PR 29572 at commit [`1a49356`](https://github.com/apache/spark/commit/1a49356b8b7e021c1c8f7176e70b3d26ce8fc491).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686001236


   **[Test build #128212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128212/testReport)** for PR 29572 at commit [`037b876`](https://github.com/apache/spark/commit/037b876bb1726d0f8a7eb6b574a0daedeb5d53e4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686469946


   **[Test build #128243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128243/testReport)** for PR 29572 at commit [`f699118`](https://github.com/apache/spark/commit/f699118df05e25193a81c9bedce5b4eb10023079).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683257761


   **[Test build #128014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128014/testReport)** for PR 29572 at commit [`acc6646`](https://github.com/apache/spark/commit/acc6646e7e83fa9e3f082b1aaa8c6227e3d8a7cf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482765771



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -662,7 +661,8 @@ private[joins] class SortMergeJoinScanner(
     bufferedIter: RowIterator,
     inMemoryThreshold: Int,
     spillThreshold: Int,
-    eagerCleanupResources: () => Unit) {
+    eagerCleanupResources: () => Unit,
+    matchedBufferFirstOnly: Boolean = false) {

Review comment:
       how about `onlyBufferFirstMatch`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486833087



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686477951






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686477951






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29572:
URL: https://github.com/apache/spark/pull/29572


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482731221



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -662,7 +668,8 @@ private[joins] class SortMergeJoinScanner(
     bufferedIter: RowIterator,
     inMemoryThreshold: Int,
     spillThreshold: Int,
-    eagerCleanupResources: () => Unit) {
+    eagerCleanupResources: () => Unit,
+    bufferFirstOnly: Boolean) {

Review comment:
       `bufferFirstOnly: Boolean = false` to avoid the unnecessary changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686179825


   **[Test build #128212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128212/testReport)** for PR 29572 at commit [`037b876`](https://github.com/apache/spark/commit/037b876bb1726d0f8a7eb6b574a0daedeb5d53e4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683129708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683286448






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686304845


   **[Test build #128238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128238/testReport)** for PR 29572 at commit [`3937e4c`](https://github.com/apache/spark/commit/3937e4ca0b8631d785543c53d492eaa5b17b1ab4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686475839


   **[Test build #128242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128242/testReport)** for PR 29572 at commit [`5cf3ab3`](https://github.com/apache/spark/commit/5cf3ab329d77b4cc525b2f4ca1992b75782a7f05).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482780387



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -156,8 +156,7 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
-            )

Review comment:
       Sorry, reverted them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-685995824






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686305492






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482322793



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArray.scala
##########
@@ -50,7 +58,7 @@ private[sql] class ExternalAppendOnlyUnsafeRowArray(
     initialSize: Int,
     pageSizeBytes: Long,
     numRowsInMemoryBufferThreshold: Int,
-    numRowsSpillThreshold: Int) extends Logging {
+    numRowsSpillThreshold: Int) extends AppendOnlyUnsafeRowArray with Logging {

Review comment:
       All right, in that case let's drop that new trait and stick to the important part.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r481958099



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,29 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
+  private[this] val bufferedMatches: AppendOnlyUnsafeRowArray = if (bufferFirstOnly) {
+    new AppendOnlyUnsafeRowArray {
+      var buffer: UnsafeRow = null
+
+      override def clear(): Unit = {
+        buffer = null
+      }
+
+      override def add(row: UnsafeRow): Unit = {
+        assert(buffer == null)

Review comment:
       No, its threshold parameters do work as expected. Just `ExternalAppendOnlyUnsafeRowArray` looked a bit heavy weight for this case when we want to store only one row. But we can also use `new ExternalAppendOnlyUnsafeRowArray(1, 1)` for this case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683257875






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686322900


   **[Test build #128243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128243/testReport)** for PR 29572 at commit [`f699118`](https://github.com/apache/spark/commit/f699118df05e25193a81c9bedce5b4eb10023079).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686180452






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r479625983



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##########
@@ -749,6 +749,15 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
         )
       }
 
+      // LEFT SEMI JOIN without bound condition does not use [[ExternalAppendOnlyUnsafeRowArray]]
+      // so should not cause any spill
+      assertNotSpilled(sparkContext, "left semi join") {

Review comment:
       Without this fix this UT fails.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683286448






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686471285






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686319217


   **[Test build #128242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128242/testReport)** for PR 29572 at commit [`5cf3ab3`](https://github.com/apache/spark/commit/5cf3ab329d77b4cc525b2f4ca1992b75782a7f05).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683286225


   **[Test build #128014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128014/testReport)** for PR 29572 at commit [`acc6646`](https://github.com/apache/spark/commit/acc6646e7e83fa9e3f082b1aaa8c6227e3d8a7cf).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683129162


   **[Test build #128000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128000/testReport)** for PR 29572 at commit [`1a49356`](https://github.com/apache/spark/commit/1a49356b8b7e021c1c8f7176e70b3d26ce8fc491).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686322689


   > Ur one more; please update the title/description (this PR is not only for leftsemi, right?), too.
   
   Updated, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686445718






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686523440


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-690678478


   > @peter-toth Nice fix! Could you share the perf difference when you run the TPC-DS ?
   
   Thanks @gatorsmile. Yes, I will try to run some benchmarks for this particular change and share the results.
   
   BTW, I have another PR open that brings ~30% improvement to some of the TPCDS queries: https://github.com/apache/spark/pull/28885


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486756148



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686471285






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683257761


   **[Test build #128014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128014/testReport)** for PR 29572 at commit [`acc6646`](https://github.com/apache/spark/commit/acc6646e7e83fa9e3f082b1aaa8c6227e3d8a7cf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686323730






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686445718






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486833087



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482776713



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -156,8 +156,7 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
-            )

Review comment:
       nit: please avoid the unnecessary changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683203012






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-684901842


   cc @cloud-fan, @maropu, @viirya 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486593079



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       Thanks @juliuszsompolski, I think you are right. Shall I open a follow-up PR or a different ticket?
   
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482774206



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -662,7 +661,8 @@ private[joins] class SortMergeJoinScanner(
     bufferedIter: RowIterator,
     inMemoryThreshold: Int,
     spillThreshold: Int,
-    eagerCleanupResources: () => Unit) {
+    eagerCleanupResources: () => Unit,
+    matchedBufferFirstOnly: Boolean = false) {

Review comment:
       Ok, renamed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683129708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686444628


   **[Test build #128238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128238/testReport)** for PR 29572 at commit [`3937e4c`](https://github.com/apache/spark/commit/3937e4ca0b8631d785543c53d492eaa5b17b1ab4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686001236


   **[Test build #128212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128212/testReport)** for PR 29572 at commit [`037b876`](https://github.com/apache/spark/commit/037b876bb1726d0f8a7eb6b574a0daedeb5d53e4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482726040



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,9 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
-    new ExternalAppendOnlyUnsafeRowArray(inMemoryThreshold, spillThreshold)
+  private[this] val bufferedMatches: ExternalAppendOnlyUnsafeRowArray =
+    new ExternalAppendOnlyUnsafeRowArray(if (bufferFirstOnly) 1 else inMemoryThreshold,

Review comment:
       how does this change avoid spilling?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486833087



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I've opened https://github.com/apache/spark/pull/29727




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486756148



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       I think a followup PR is fine.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482755812



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -662,7 +668,8 @@ private[joins] class SortMergeJoinScanner(
     bufferedIter: RowIterator,
     inMemoryThreshold: Int,
     spillThreshold: Int,
-    eagerCleanupResources: () => Unit) {
+    eagerCleanupResources: () => Unit,
+    bufferFirstOnly: Boolean) {

Review comment:
       Thanks, fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482002324



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,29 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
+  private[this] val bufferedMatches: AppendOnlyUnsafeRowArray = if (bufferFirstOnly) {
+    new AppendOnlyUnsafeRowArray {
+      var buffer: UnsafeRow = null
+
+      override def clear(): Unit = {
+        buffer = null
+      }
+
+      override def add(row: UnsafeRow): Unit = {
+        assert(buffer == null)

Review comment:
       How did you test it if `ExternalAppendOnlyUnsafeRowArray` doesn't spill either?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-685995824






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-689256055


   @peter-toth Nice fix! Could you share the perf difference when you run the TPC-DS ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686318625


   Ur one more; please update the title/description (this PR is not only for leftsemi, right?), too.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] juliuszsompolski commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
juliuszsompolski commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r486490381



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -251,7 +251,8 @@ case class SortMergeJoinExec(
               RowIterator.fromScala(rightIter),
               inMemoryThreshold,
               spillThreshold,
-              cleanupResources
+              cleanupResources,
+              condition.isEmpty

Review comment:
       Thanks @peter-toth !
   I think this could be also added to LeftAnti join, which is also only interested in the existence of a match and doesn't need to buffer them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683203012






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29572: [WIP][SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683202640


   **[Test build #128000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128000/testReport)** for PR 29572 at commit [`1a49356`](https://github.com/apache/spark/commit/1a49356b8b7e021c1c8f7176e70b3d26ce8fc491).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `trait AppendOnlyUnsafeRowArray `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-683257875






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] peter-toth commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482753704



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,9 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
-    new ExternalAppendOnlyUnsafeRowArray(inMemoryThreshold, spillThreshold)
+  private[this] val bufferedMatches: ExternalAppendOnlyUnsafeRowArray =
+    new ExternalAppendOnlyUnsafeRowArray(if (bufferFirstOnly) 1 else inMemoryThreshold,

Review comment:
       There is another call site in existence join: https://github.com/apache/spark/pull/29572/files#diff-2649ee0724bbc5ee6a64d7de308c527eR336




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r481907082



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,29 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
+  private[this] val bufferedMatches: AppendOnlyUnsafeRowArray = if (bufferFirstOnly) {
+    new AppendOnlyUnsafeRowArray {
+      var buffer: UnsafeRow = null
+
+      override def clear(): Unit = {
+        buffer = null
+      }
+
+      override def add(row: UnsafeRow): Unit = {
+        assert(buffer == null)

Review comment:
       Are you saying that `ExternalAppendOnlyUnsafeRowArray` will do spill even if we only add one row?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686322900


   **[Test build #128243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128243/testReport)** for PR 29572 at commit [`f699118`](https://github.com/apache/spark/commit/f699118df05e25193a81c9bedce5b4eb10023079).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29572:
URL: https://github.com/apache/spark/pull/29572#issuecomment-686305492






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482731864



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##########
@@ -673,8 +680,9 @@ private[joins] class SortMergeJoinScanner(
    */
   private[this] var matchJoinKey: InternalRow = _
   /** Buffered rows from the buffered side of the join. This is empty if there are no matches. */
-  private[this] val bufferedMatches =
-    new ExternalAppendOnlyUnsafeRowArray(inMemoryThreshold, spillThreshold)
+  private[this] val bufferedMatches: ExternalAppendOnlyUnsafeRowArray =
+    new ExternalAppendOnlyUnsafeRowArray(if (bufferFirstOnly) 1 else inMemoryThreshold,

Review comment:
       We need this change? How about just setting 1 in the caller side? https://github.com/apache/spark/pull/29572/files#diff-2649ee0724bbc5ee6a64d7de308c527eR255




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #29572: [SPARK-32730][SQL] Improve LeftSemi SortMergeJoin right side buffering

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29572:
URL: https://github.com/apache/spark/pull/29572#discussion_r482198804



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArray.scala
##########
@@ -50,7 +58,7 @@ private[sql] class ExternalAppendOnlyUnsafeRowArray(
     initialSize: Int,
     pageSizeBytes: Long,
     numRowsInMemoryBufferThreshold: Int,
-    numRowsSpillThreshold: Int) extends Logging {
+    numRowsSpillThreshold: Int) extends AppendOnlyUnsafeRowArray with Logging {

Review comment:
       For core component like this, I remember we rarely change its inheritance. It is easily to have performance regression.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org