You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/27 14:50:03 UTC

[GitHub] [spark] tanelk opened a new pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

tanelk opened a new pull request #34402:
URL: https://github.com/apache/spark/pull/34402


   ### What changes were proposed in this pull request?
   Enable using Exists/In subqueries in other nodes besides `Filter`. For example `Aggregate`, `Project`, `Window`, etc.
   This is allready mostly supported, but it was blocked in the `Analyzer`. Only requires a small tweak in the `Optimizer`.
   
   ### Why are the changes needed?
   One of the last open feature parities between PostgreSQL and Spark: [SPARK-30374](https://issues.apache.org/jira/browse/SPARK-30374)
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   Existing and new UTs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962895478


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49456/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954781035


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49225/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954544458


   **[Test build #144737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144737/testReport)** for PR 34402 at commit [`0277dcc`](https://github.com/apache/spark/commit/0277dcc911b12b09f313af8f8b21de4260a7ef88).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414


   **[Test build #144833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144833/testReport)** for PR 34402 at commit [`ce3adb5`](https://github.com/apache/spark/commit/ce3adb5c7f148c71e222027661335d458b997534).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740540249



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       How to make sure that all UnaryNode can support IN/EXISTS subqueries (correlated and uncorrelated)? The test cases only covered Project, Aggregate, and Window. Maybe we could either only support these three nodes for now, or add more tests to cover other unary operators.
   
   Also, would it work with Aggregate having correlated IN/EXISTS subqueries in the GROUP BY clause?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953191319


   **[Test build #144664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144664/testReport)** for PR 34402 at commit [`65c0907`](https://github.com/apache/spark/commit/65c09070e8ce8dafbe65c455fc95e58a68457dc0).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953027217


   **[Test build #144664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144664/testReport)** for PR 34402 at commit [`65c0907`](https://github.com/apache/spark/commit/65c09070e8ce8dafbe65c455fc95e58a68457dc0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954702739


   **[Test build #144756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144756/testReport)** for PR 34402 at commit [`6ba940a`](https://github.com/apache/spark/commit/6ba940a599b967e7faf9a338a8306e9cee0e65c2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957153269


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49303/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957133911


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49303/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829617806



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       If so we will also need to check the ScalarSubquery checks above:
   https://github.com/apache/spark/blob/f36a5fb2b88620c1c490d087b0293c4e58d29979/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L743-L747




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829617806



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       If so we will also need to check the ScalarSubquery checks above:
   https://github.com/apache/spark/blob/f36a5fb2b88620c1c490d087b0293c4e58d29979/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L743-L747




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953192007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144664/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954545332


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144737/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954964257


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144758/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962859629


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49456/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962895478


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49456/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829967745



##########
File path: sql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
##########
@@ -0,0 +1,133 @@
+-- Tests EXISTS subquery support where the subquery is used outside the WHERE clause.
+
+
+CREATE TEMPORARY VIEW EMP AS SELECT * FROM VALUES
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (200, "emp 2", date "2003-01-01", 200.00D, 10),
+  (300, "emp 3", date "2002-01-01", 300.00D, 20),
+  (400, "emp 4", date "2005-01-01", 400.00D, 30),
+  (500, "emp 5", date "2001-01-01", 400.00D, NULL),
+  (600, "emp 6 - no dept", date "2001-01-01", 400.00D, 100),
+  (700, "emp 7", date "2010-01-01", 400.00D, 100),
+  (800, "emp 8", date "2016-01-01", 150.00D, 70)
+AS EMP(id, emp_name, hiredate, salary, dept_id);
+
+CREATE TEMPORARY VIEW DEPT AS SELECT * FROM VALUES
+  (10, "dept 1", "CA"),
+  (20, "dept 2", "NY"),
+  (30, "dept 3", "TX"),
+  (40, "dept 4 - unassigned", "OR"),
+  (50, "dept 5 - unassigned", "NJ"),
+  (70, "dept 7", "FL")
+AS DEPT(dept_id, dept_name, state);
+
+CREATE TEMPORARY VIEW BONUS AS SELECT * FROM VALUES
+  ("emp 1", 10.00D),
+  ("emp 1", 20.00D),
+  ("emp 2", 300.00D),
+  ("emp 2", 100.00D),
+  ("emp 3", 300.00D),
+  ("emp 4", 100.00D),
+  ("emp 5", 1000.00D),
+  ("emp 6 - no dept", 500.00D)
+AS BONUS(emp_name, bonus_amt);
+
+-- uncorrelated select exist
+-- TC.01.01
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  dept.dept_id > 10
+            AND dept.dept_id < 30)
+FROM   emp;
+
+-- correlated select exist
+-- TC.01.02
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  emp.dept_id = dept.dept_id)
+FROM   emp;
+
+-- uncorrelated exist in aggregate filter

Review comment:
       Added testcases

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -658,16 +658,6 @@ class AnalysisErrorSuite extends AnalysisTest {
     assertAnalysisError(plan2, "EqualTo does not support ordering on type map" :: Nil)
   }
 
-  test("PredicateSubQuery is used outside of a filter") {

Review comment:
       Restrored it

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       ```
   /**
    * A trait to represent the commands that support subqueries.
    * This is used to allow such commands in the subquery-related checks.
    */
   trait SupportsSubquery extends LogicalPlan
   ```
   
   Yeah, I'm bit sceptical whit this one. It states, that it works on commands.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       ```
   /**
    * A trait to represent the commands that support subqueries.
    * This is used to allow such commands in the subquery-related checks.
    */
   trait SupportsSubquery extends LogicalPlan
   ```
   
   Yeah, I'm bit sceptical whit this one. It states, that it works on commands. Not sure what kind of sideeffects it can have.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954749086


   **[Test build #144758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144758/testReport)** for PR 34402 at commit [`1705630`](https://github.com/apache/spark/commit/170563055744d6d1965876d862bb2da717d9bd52).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954780986


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49225/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954749086


   **[Test build #144758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144758/testReport)** for PR 34402 at commit [`1705630`](https://github.com/apache/spark/commit/170563055744d6d1965876d862bb2da717d9bd52).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954545332


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144737/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962834859


   **[Test build #144985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144985/testReport)** for PR 34402 at commit [`0369168`](https://github.com/apache/spark/commit/0369168b75341ce7dc562ad8889875732c3f4007).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r743992678



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       I think it's safer to use a whitelist of nodes than to support all UnaryNodes, and it shouldn't be a common use case for EXISTS/IN subquery to be in Generate, Expand, Limit, etc. We can always add more nodes in the future if needed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829117450



##########
File path: sql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
##########
@@ -0,0 +1,133 @@
+-- Tests EXISTS subquery support where the subquery is used outside the WHERE clause.
+
+
+CREATE TEMPORARY VIEW EMP AS SELECT * FROM VALUES
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (200, "emp 2", date "2003-01-01", 200.00D, 10),
+  (300, "emp 3", date "2002-01-01", 300.00D, 20),
+  (400, "emp 4", date "2005-01-01", 400.00D, 30),
+  (500, "emp 5", date "2001-01-01", 400.00D, NULL),
+  (600, "emp 6 - no dept", date "2001-01-01", 400.00D, 100),
+  (700, "emp 7", date "2010-01-01", 400.00D, 100),
+  (800, "emp 8", date "2016-01-01", 150.00D, 70)
+AS EMP(id, emp_name, hiredate, salary, dept_id);
+
+CREATE TEMPORARY VIEW DEPT AS SELECT * FROM VALUES
+  (10, "dept 1", "CA"),
+  (20, "dept 2", "NY"),
+  (30, "dept 3", "TX"),
+  (40, "dept 4 - unassigned", "OR"),
+  (50, "dept 5 - unassigned", "NJ"),
+  (70, "dept 7", "FL")
+AS DEPT(dept_id, dept_name, state);
+
+CREATE TEMPORARY VIEW BONUS AS SELECT * FROM VALUES
+  ("emp 1", 10.00D),
+  ("emp 1", 20.00D),
+  ("emp 2", 300.00D),
+  ("emp 2", 100.00D),
+  ("emp 3", 300.00D),
+  ("emp 4", 100.00D),
+  ("emp 5", 1000.00D),
+  ("emp 6 - no dept", 500.00D)
+AS BONUS(emp_name, bonus_amt);
+
+-- uncorrelated select exist
+-- TC.01.01
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  dept.dept_id > 10
+            AND dept.dept_id < 30)
+FROM   emp;
+
+-- correlated select exist
+-- TC.01.02
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  emp.dept_id = dept.dept_id)
+FROM   emp;
+
+-- uncorrelated exist in aggregate filter

Review comment:
       do we support subqueries in aggregate function inputs?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829113889



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       shall we simply make these plan nodes extend `SupportsSubquery`?

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -658,16 +658,6 @@ class AnalysisErrorSuite extends AnalysisTest {
     assertAnalysisError(plan2, "EqualTo does not support ordering on type map" :: Nil)
   }
 
-  test("PredicateSubQuery is used outside of a filter") {

Review comment:
       can we still keep this test by putting subquery in sort?

##########
File path: sql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
##########
@@ -0,0 +1,133 @@
+-- Tests EXISTS subquery support where the subquery is used outside the WHERE clause.
+
+
+CREATE TEMPORARY VIEW EMP AS SELECT * FROM VALUES
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (200, "emp 2", date "2003-01-01", 200.00D, 10),
+  (300, "emp 3", date "2002-01-01", 300.00D, 20),
+  (400, "emp 4", date "2005-01-01", 400.00D, 30),
+  (500, "emp 5", date "2001-01-01", 400.00D, NULL),
+  (600, "emp 6 - no dept", date "2001-01-01", 400.00D, 100),
+  (700, "emp 7", date "2010-01-01", 400.00D, 100),
+  (800, "emp 8", date "2016-01-01", 150.00D, 70)
+AS EMP(id, emp_name, hiredate, salary, dept_id);
+
+CREATE TEMPORARY VIEW DEPT AS SELECT * FROM VALUES
+  (10, "dept 1", "CA"),
+  (20, "dept 2", "NY"),
+  (30, "dept 3", "TX"),
+  (40, "dept 4 - unassigned", "OR"),
+  (50, "dept 5 - unassigned", "NJ"),
+  (70, "dept 7", "FL")
+AS DEPT(dept_id, dept_name, state);
+
+CREATE TEMPORARY VIEW BONUS AS SELECT * FROM VALUES
+  ("emp 1", 10.00D),
+  ("emp 1", 20.00D),
+  ("emp 2", 300.00D),
+  ("emp 2", 100.00D),
+  ("emp 3", 300.00D),
+  ("emp 4", 100.00D),
+  ("emp 5", 1000.00D),
+  ("emp 6 - no dept", 500.00D)
+AS BONUS(emp_name, bonus_amt);
+
+-- uncorrelated select exist
+-- TC.01.01
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  dept.dept_id > 10
+            AND dept.dept_id < 30)
+FROM   emp;
+
+-- correlated select exist
+-- TC.01.02
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  emp.dept_id = dept.dept_id)
+FROM   emp;
+
+-- uncorrelated exist in aggregate filter

Review comment:
       do we support subqueries in aggregate function inputs?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953110972


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49133/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953027217


   **[Test build #144664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144664/testReport)** for PR 34402 at commit [`65c0907`](https://github.com/apache/spark/commit/65c09070e8ce8dafbe65c455fc95e58a68457dc0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954964257


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144758/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414


   **[Test build #144833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144833/testReport)** for PR 34402 at commit [`ce3adb5`](https://github.com/apache/spark/commit/ce3adb5c7f148c71e222027661335d458b997534).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740540249



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       How to make sure that all UnaryNode can support IN/EXISTS subqueries (correlated and uncorrelated)? The test cases only covered Project, Aggregate, and Window. Maybe we could either only support these three nodes for now, or add more tests to cover other unary operators.
   
   Also, would it work with Aggregate having correlated IN/EXISTS subqueries in the GROUP BY clause?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953131688


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49133/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954736574


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49225/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954826059


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49227/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954488735


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49206/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954918410


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144756/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954918410


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144756/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-1070391108


   @HyukjinKwon could you take a look at this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829969051



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       ```
   /**
    * A trait to represent the commands that support subqueries.
    * This is used to allow such commands in the subquery-related checks.
    */
   trait SupportsSubquery extends LogicalPlan
   ```
   
   Yeah, I'm bit sceptical whit this one. It states, that it works on commands. Not sure what kind of sideeffects it can have.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829115538



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -658,16 +658,6 @@ class AnalysisErrorSuite extends AnalysisTest {
     assertAnalysisError(plan2, "EqualTo does not support ordering on type map" :: Nil)
   }
 
-  test("PredicateSubQuery is used outside of a filter") {

Review comment:
       can we still keep this test by putting subquery in sort?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953131688


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49133/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954437674


   **[Test build #144737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144737/testReport)** for PR 34402 at commit [`0277dcc`](https://github.com/apache/spark/commit/0277dcc911b12b09f313af8f8b21de4260a7ef88).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957245805


   **[Test build #144833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144833/testReport)** for PR 34402 at commit [`ce3adb5`](https://github.com/apache/spark/commit/ce3adb5c7f148c71e222027661335d458b997534).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49303/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954821347


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49227/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954826059


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49227/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414


   **[Test build #144833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144833/testReport)** for PR 34402 at commit [`ce3adb5`](https://github.com/apache/spark/commit/ce3adb5c7f148c71e222027661335d458b997534).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740718765



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       The Exists/In subqueries in UnaryNodes are handled by inserting an existance join (or joins) bellow the UnaryNode and then the subquery is replaced with an attribute reference. This is a very generic pattern and should not depend on the actual UnaryNode. Yes, Filter is handled separately but that is a performance optimization, not a correctness issue (LeftSemi/LeftAnti are better than existance join).
   
   But if you think that we should stay safe and only allow a subset of UnaryNodes, then I would be totally fine with supporting only the most common nodes - Project, Aggregate, Window and perhaps Sort.
   
   I'll add some extra test cases for aggregate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-963027061


   **[Test build #144985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144985/testReport)** for PR 34402 at commit [`0369168`](https://github.com/apache/spark/commit/0369168b75341ce7dc562ad8889875732c3f4007).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-1073777556


   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34402:
URL: https://github.com/apache/spark/pull/34402


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-963032634


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144985/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49303/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953071762


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49133/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954781035


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49225/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954437674


   **[Test build #144737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144737/testReport)** for PR 34402 at commit [`0277dcc`](https://github.com/apache/spark/commit/0277dcc911b12b09f313af8f8b21de4260a7ef88).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954702739


   **[Test build #144756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144756/testReport)** for PR 34402 at commit [`6ba940a`](https://github.com/apache/spark/commit/6ba940a599b967e7faf9a338a8306e9cee0e65c2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957264007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144833/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954483467


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49206/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954488735


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49206/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414


   **[Test build #144833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144833/testReport)** for PR 34402 at commit [`ce3adb5`](https://github.com/apache/spark/commit/ce3adb5c7f148c71e222027661335d458b997534).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-953192007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144664/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [WIP][SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954456277


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49206/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954963166


   **[Test build #144758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144758/testReport)** for PR 34402 at commit [`1705630`](https://github.com/apache/spark/commit/170563055744d6d1965876d862bb2da717d9bd52).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962834859


   **[Test build #144985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144985/testReport)** for PR 34402 at commit [`0369168`](https://github.com/apache/spark/commit/0369168b75341ce7dc562ad8889875732c3f4007).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-963032634


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144985/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829967745



##########
File path: sql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
##########
@@ -0,0 +1,133 @@
+-- Tests EXISTS subquery support where the subquery is used outside the WHERE clause.
+
+
+CREATE TEMPORARY VIEW EMP AS SELECT * FROM VALUES
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (100, "emp 1", date "2005-01-01", 100.00D, 10),
+  (200, "emp 2", date "2003-01-01", 200.00D, 10),
+  (300, "emp 3", date "2002-01-01", 300.00D, 20),
+  (400, "emp 4", date "2005-01-01", 400.00D, 30),
+  (500, "emp 5", date "2001-01-01", 400.00D, NULL),
+  (600, "emp 6 - no dept", date "2001-01-01", 400.00D, 100),
+  (700, "emp 7", date "2010-01-01", 400.00D, 100),
+  (800, "emp 8", date "2016-01-01", 150.00D, 70)
+AS EMP(id, emp_name, hiredate, salary, dept_id);
+
+CREATE TEMPORARY VIEW DEPT AS SELECT * FROM VALUES
+  (10, "dept 1", "CA"),
+  (20, "dept 2", "NY"),
+  (30, "dept 3", "TX"),
+  (40, "dept 4 - unassigned", "OR"),
+  (50, "dept 5 - unassigned", "NJ"),
+  (70, "dept 7", "FL")
+AS DEPT(dept_id, dept_name, state);
+
+CREATE TEMPORARY VIEW BONUS AS SELECT * FROM VALUES
+  ("emp 1", 10.00D),
+  ("emp 1", 20.00D),
+  ("emp 2", 300.00D),
+  ("emp 2", 100.00D),
+  ("emp 3", 300.00D),
+  ("emp 4", 100.00D),
+  ("emp 5", 1000.00D),
+  ("emp 6 - no dept", 500.00D)
+AS BONUS(emp_name, bonus_amt);
+
+-- uncorrelated select exist
+-- TC.01.01
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  dept.dept_id > 10
+            AND dept.dept_id < 30)
+FROM   emp;
+
+-- correlated select exist
+-- TC.01.02
+SELECT
+  emp_name,
+  EXISTS (SELECT 1
+          FROM   dept
+          WHERE  emp.dept_id = dept.dept_id)
+FROM   emp;
+
+-- uncorrelated exist in aggregate filter

Review comment:
       Added testcases




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954789349


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49227/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954916848


   **[Test build #144756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144756/testReport)** for PR 34402 at commit [`6ba940a`](https://github.com/apache/spark/commit/6ba940a599b967e7faf9a338a8306e9cee0e65c2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-954974929


   pinging @viirya , @cloud-fan 
   Could you take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740718765



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       The Exists/In subqueries in UnaryNodes are handled by inserting an existance join (or joins) bellow the UnaryNode and then the subquery is replaced with an attribute reference. This is a very generic pattern and should not depend on the actual UnaryNode. Yes, Filter is handled separately but that is a performance optimization, not a correctness issue (LeftSemi/LeftAnti are better than existance join).
   
   But if you think that we should stay safe and only allow a subset of UnaryNodes, then I would be totally fine with supporting only the most common nodes - Project, Aggregate, Window and perhaps Sort.
   
   I'll add some extra test cases for aggregate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740718765



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       The Exists/In subqueries in UnaryNodes are handled by inserting an existance join (or joins) bellow the UnaryNode and then the subquery is replaced with an attribute reference. This is a very generic pattern and should not depend on the actual UnaryNode. Yes, Filter is handled separately but that is a performance optimization, not a correctness issue (LeftSemi/LeftAnti are better than existance join).
   
   But if you think that we should stay safe and only allow a subset of UnaryNodes, then I would be totally fine with supporting only the most common nodes - Project, Aggregate, Window and perhaps Sort.
   
   I'll add some extra test cases for aggregate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957164941






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957114414






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-957264007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144833/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r740540249



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -687,10 +687,10 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: SupportsSubquery | _: Join | _: UnaryNode => // Ok

Review comment:
       How to make sure that all UnaryNode can support IN/EXISTS subqueries (correlated and uncorrelated)? The test cases only covered Project, Aggregate, and Window. Maybe we could either only support these three nodes for now, or add more tests to cover other unary operators.
   
   Also, would it work with Aggregate having correlated IN/EXISTS subqueries in the GROUP BY clause?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] allisonwang-db commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962239890


   cc @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34402:
URL: https://github.com/apache/spark/pull/34402#issuecomment-962885744


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49456/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829113889



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       shall we simply make these plan nodes extend `SupportsSubquery`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829969051



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -782,10 +782,11 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
 
       case inSubqueryOrExistsSubquery =>
         plan match {
-          case _: Filter | _: SupportsSubquery | _: Join => // Ok
+          case _: Filter | _: SupportsSubquery | _: Join |
+            _: Project | _: Aggregate | _: Window => // Ok

Review comment:
       ```
   /**
    * A trait to represent the commands that support subqueries.
    * This is used to allow such commands in the subquery-related checks.
    */
   trait SupportsSubquery extends LogicalPlan
   ```
   
   Yeah, I'm bit sceptical whit this one. It states, that it works on commands.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #34402: [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #34402:
URL: https://github.com/apache/spark/pull/34402#discussion_r829968135



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -658,16 +658,6 @@ class AnalysisErrorSuite extends AnalysisTest {
     assertAnalysisError(plan2, "EqualTo does not support ordering on type map" :: Nil)
   }
 
-  test("PredicateSubQuery is used outside of a filter") {

Review comment:
       Restrored it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org