You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/04 22:24:28 UTC
[GitHub] [spark] allisonwang-db opened a new pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
allisonwang-db opened a new pull request #32787:
URL: https://github.com/apache/spark/pull/32787
### What changes were proposed in this pull request?
This PR supports resolving star expressions in subqueries using outer query plans.
### Why are the changes needed?
Currently, Spark can only resolve star expressions using the inner query plan when resolving subqueries. Instead, it should also be able to resolve star expressions using the outer query plans.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870980294
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44904/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868762786
**[Test build #140339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140339/testReport)** for PR 32787 at commit [`b093b97`](https://github.com/apache/spark/commit/b093b97e98668cecd7f0cf52ccc830c158e0b22c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r657816274
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1571,6 +1582,26 @@ class Analyzer(override val catalogManager: CatalogManager)
}
}
+ // Expand the star expression using the input plan first. If failed, try resolve
+ // the star expression using the outer query plan and wrap the resolved attributes
+ // in outer references. Otherwise throw the original exception.
+ private def expand(s: Star, plan: LogicalPlan): Seq[NamedExpression] = {
+ withPosition(s) {
+ try {
+ s.expand(plan, resolver)
+ } catch {
+ case e: AnalysisException =>
+ AnalysisContext.get.outerPlan.map(p =>
+ // Only a few unary nodes (Project/Aggregate) can host star expressions.
+ Try(s.expand(p.children.head, resolver)) match {
Review comment:
how do we expand star for join in the main query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r660349216
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -807,4 +807,48 @@ class AnalysisErrorSuite extends AnalysisTest {
// UnresolvedHint be removed by batch `Remove Unresolved Hints`
assertAnalysisSuccess(plan, true)
}
+
+ test("SPARK-35618: Resolve star expressions in subqueries") {
+ val a = AttributeReference("a", IntegerType)()
+ val b = AttributeReference("b", IntegerType)()
+ val t0 = OneRowRelation()
+ val t1 = LocalRelation(a, b).as("t1")
+
+ // t1.* in the subquery should be resolved into outer(t1.a) and outer(t1.b).
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(star("t1"))).as("sub") :: Nil, t1),
+ "Scalar subquery must return only one column, but got 2" :: Nil)
+
+ // array(t1.*) in the subquery should be resolved into array(outer(t1.a), outer(t1.b))
+ val array = CreateArray(Seq(star("t1")))
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(array)).as("sub") :: Nil, t1),
+ "Expressions referencing the outer query are not supported" :: Nil)
Review comment:
hmm, I think we do support star in `CreateArray`? https://github.com/apache/spark/pull/32787/files#diff-c00a384380a1462ecad9fca0fed853c47d6894867420f7daf94b1c56ae444e4bR226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872080973
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867252043
**[Test build #140221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140221/testReport)** for PR 32787 at commit [`8f1f42a`](https://github.com/apache/spark/commit/8f1f42a134901524872ada157c82215313d03e7d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r660149328
##########
File path: sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out
##########
@@ -80,6 +80,46 @@ struct<c1:int,c2:int,c1:int,c2:int>
1 2 0 3
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1
+1 2 1 2
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1 0 2
+0 1 0 1 0 3
+1 2 1 2 0 2
+1 2 1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.* FROM t2 AS t1)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 2
+0 1 0 3
+1 2 0 2
+1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3))
Review comment:
Good question. Current Spark can only resolve the subquery using the immediate outer query. In this case t1.* can only be resolved using output from t2, which is c1, c2. There are many other places need to be updated to support nested subqueries with deep correlations.
https://github.com/apache/spark/blob/9c157a490bb59e02dcf44b14b411ea5beb68c238/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala#L164-L173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856347555
**[Test build #139437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139437/testReport)** for PR 32787 at commit [`05b0613`](https://github.com/apache/spark/commit/05b061361b2e7259d8d5d7a8cb7b9b9101bdb0c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862174258
**[Test build #139855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139855/testReport)** for PR 32787 at commit [`e0460c5`](https://github.com/apache/spark/commit/e0460c52c8d0d99eb06a618228ff7cd51b9c97ab).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856440257
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r652363922
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala
##########
@@ -177,4 +178,61 @@ class ResolveSubquerySuite extends AnalysisTest {
condition = Some(sum('a) === sum('c)))
assertAnalysisError(plan, Seq("Invalid expressions: [sum(a), sum(c)]"))
}
+
+ test("SPARK-35618: lateral join with star expansion") {
Review comment:
@maropu I looked into how regex expressions are resolved and the logic is actually different from star expressions. It won't throw exceptions when there is no match. Instead, it returns an empty sequence. So we can't tell if the regex expression is resolved by the current plan with an empty output, or it can't be resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r658953235
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1571,6 +1582,26 @@ class Analyzer(override val catalogManager: CatalogManager)
}
}
+ // Expand the star expression using the input plan first. If failed, try resolve
+ // the star expression using the outer query plan and wrap the resolved attributes
+ // in outer references. Otherwise throw the original exception.
+ private def expand(s: Star, plan: LogicalPlan): Seq[NamedExpression] = {
+ withPosition(s) {
+ try {
+ s.expand(plan, resolver)
+ } catch {
+ case e: AnalysisException =>
+ AnalysisContext.get.outerPlan.map(p =>
+ // Only a few unary nodes (Project/Aggregate) can host star expressions.
+ Try(s.expand(p.children.head, resolver)) match {
Review comment:
The error message is confusing when there are star usages other than Project and Filter. Will update CheckAnslysis.
```
scala> sql("select * from t1 join t2 on t1.* = t2.c1")
org.apache.spark.sql.AnalysisException: Invalid call to dataType on unresolved object;
'Project [*]
+- 'Join Inner, (ArrayBuffer(t1).* = c1#76)
:- SubqueryAlias spark_catalog.default.t1
: +- View (`default`.`t1`, [c1#72,c2#73])
: +- Project [cast(col1#74 as int) AS c1#72, cast(col2#75 as int) AS c2#73]
: +- LocalRelation [col1#74, col2#75]
+- SubqueryAlias spark_catalog.default.t2
+- View (`default`.`t2`, [c1#76,c2#77])
+- Project [cast(col1#78 as int) AS c1#76, cast(col2#79 as int) AS c2#77]
+- LocalRelation [col1#78, col2#79]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-871969800
**[Test build #140491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140491/testReport)** for PR 32787 at commit [`0346d3b`](https://github.com/apache/spark/commit/0346d3b7c0cbbe3edf88a1451da9bf53726a458d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868805691
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872133337
**[Test build #140491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140491/testReport)** for PR 32787 at commit [`0346d3b`](https://github.com/apache/spark/commit/0346d3b7c0cbbe3edf88a1451da9bf53726a458d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858289714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867264899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140221/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868821648
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858258406
**[Test build #139601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139601/testReport)** for PR 32787 at commit [`1704a9a`](https://github.com/apache/spark/commit/1704a9a8c54fab6b0b450c4fe32be444dc81df20).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856362112
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870933104
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858184907
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867176608
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44749/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r647164105
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -791,4 +791,28 @@ class AnalysisErrorSuite extends AnalysisTest {
assertAnalysisError(plan, s"Correlated column is not allowed in predicate ($msg)" :: Nil)
}
}
+
+ test("SPARK-35618: Resolve star expressions in subquery") {
Review comment:
Yes, currently only `Filter` can host outer references for correlated subqueries, and star expansion only happens when the node is either a `Project` or `Aggregate` (buildExpandedProjectList). It will be clearer with lateral subquery examples:
```sql
// t: [a, b]
SELECT * FROM t, LATERAL (SELECT t.*) // <--- t.* should be resolved as t.a, t.b
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870985363
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44904/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862217075
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r660149328
##########
File path: sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out
##########
@@ -80,6 +80,46 @@ struct<c1:int,c2:int,c1:int,c2:int>
1 2 0 3
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1
+1 2 1 2
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1 0 2
+0 1 0 1 0 3
+1 2 1 2 0 2
+1 2 1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.* FROM t2 AS t1)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 2
+0 1 0 3
+1 2 0 2
+1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3))
Review comment:
Good question. Current Spark can only resolve the subquery using the immediate outer query. In this case t1.* can only be resolved using output from t2, which is c1, c2. There are many other places need to be updated to support nested subqueries with deep correlations.
https://github.com/apache/spark/blob/9c157a490bb59e02dcf44b14b411ea5beb68c238/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala#L164-L173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858189868
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868885472
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140339/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872134433
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140491/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858189868
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856372389
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862186559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872086013
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r658495078
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1571,6 +1582,26 @@ class Analyzer(override val catalogManager: CatalogManager)
}
}
+ // Expand the star expression using the input plan first. If failed, try resolve
+ // the star expression using the outer query plan and wrap the resolved attributes
+ // in outer references. Otherwise throw the original exception.
+ private def expand(s: Star, plan: LogicalPlan): Seq[NamedExpression] = {
+ withPosition(s) {
+ try {
+ s.expand(plan, resolver)
+ } catch {
+ case e: AnalysisException =>
+ AnalysisContext.get.outerPlan.map(p =>
+ // Only a few unary nodes (Project/Aggregate) can host star expressions.
+ Try(s.expand(p.children.head, resolver)) match {
Review comment:
Then can we add checks here and give a proper error message if the outer plan can't host star?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867264899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140221/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855152592
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867176583
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44749/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870933104
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867140693
**[Test build #140221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140221/testReport)** for PR 32787 at commit [`8f1f42a`](https://github.com/apache/spark/commit/8f1f42a134901524872ada157c82215313d03e7d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855171582
**[Test build #139356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139356/testReport)** for PR 32787 at commit [`6150cc4`](https://github.com/apache/spark/commit/6150cc4cd570ea0f8023334a2fa8149ca7b997ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862107995
**[Test build #139855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139855/testReport)** for PR 32787 at commit [`e0460c5`](https://github.com/apache/spark/commit/e0460c52c8d0d99eb06a618228ff7cd51b9c97ab).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856372399
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872091206
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855152606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r647115051
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -791,4 +791,28 @@ class AnalysisErrorSuite extends AnalysisTest {
assertAnalysisError(plan, s"Correlated column is not allowed in predicate ($msg)" :: Nil)
}
}
+
+ test("SPARK-35618: Resolve star expressions in subquery") {
Review comment:
I read the PR description and I thought this PR is to accept new query patterns, but this PR only has the negative test cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862186559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r659464716
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1571,6 +1582,30 @@ class Analyzer(override val catalogManager: CatalogManager)
}
}
+ // Expand the star expression using the input plan first. If failed, try resolve
+ // the star expression using the outer query plan and wrap the resolved attributes
+ // in outer references. Otherwise throw the original exception.
+ private def expand(s: Star, plan: LogicalPlan): Seq[NamedExpression] = {
+ withPosition(s) {
+ try {
+ s.expand(plan, resolver)
+ } catch {
+ case e: AnalysisException =>
+ AnalysisContext.get.outerPlan.map {
+ // Only a few unary nodes (Project/Aggregate) can host star expressions.
+ case u: UnaryNode =>
Review comment:
we cannot filter project/aggregate nodes here?
##########
File path: sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out
##########
@@ -80,6 +80,46 @@ struct<c1:int,c2:int,c1:int,c2:int>
1 2 0 3
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1
+1 2 1 2
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1 0 2
+0 1 0 1 0 3
+1 2 1 2 0 2
+1 2 1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.* FROM t2 AS t1)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 2
+0 1 0 3
+1 2 0 2
+1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3))
Review comment:
Q: We cannot accept this query? It seems pogresql can though;
```
postgres=# SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3) t) t;
c1 | c2 | c1 | c2 | c1 | c2
----+----+----+----+----+----
0 | 1 | 0 | 1 | 0 | 2
0 | 1 | 0 | 1 | 0 | 2
0 | 1 | 0 | 1 | 0 | 3
0 | 1 | 0 | 1 | 0 | 3
1 | 2 | 1 | 2 | 0 | 2
1 | 2 | 1 | 2 | 0 | 2
1 | 2 | 1 | 2 | 0 | 3
1 | 2 | 1 | 2 | 0 | 3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868885472
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140339/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855048367
**[Test build #139356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139356/testReport)** for PR 32787 at commit [`6150cc4`](https://github.com/apache/spark/commit/6150cc4cd570ea0f8023334a2fa8149ca7b997ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r658450441
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -1571,6 +1582,26 @@ class Analyzer(override val catalogManager: CatalogManager)
}
}
+ // Expand the star expression using the input plan first. If failed, try resolve
+ // the star expression using the outer query plan and wrap the resolved attributes
+ // in outer references. Otherwise throw the original exception.
+ private def expand(s: Star, plan: LogicalPlan): Seq[NamedExpression] = {
+ withPosition(s) {
+ try {
+ s.expand(plan, resolver)
+ } catch {
+ case e: AnalysisException =>
+ AnalysisContext.get.outerPlan.map(p =>
+ // Only a few unary nodes (Project/Aggregate) can host star expressions.
+ Try(s.expand(p.children.head, resolver)) match {
Review comment:
Join can't host star expressions I think. Only star expressions in Project and Aggregate are handled in the Analyzer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855173701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856347555
**[Test build #139437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139437/testReport)** for PR 32787 at commit [`05b0613`](https://github.com/apache/spark/commit/05b061361b2e7259d8d5d7a8cb7b9b9101bdb0c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856429384
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139437/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868788276
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-862107995
**[Test build #139855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139855/testReport)** for PR 32787 at commit [`e0460c5`](https://github.com/apache/spark/commit/e0460c52c8d0d99eb06a618228ff7cd51b9c97ab).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856428690
**[Test build #139437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139437/testReport)** for PR 32787 at commit [`05b0613`](https://github.com/apache/spark/commit/05b061361b2e7259d8d5d7a8cb7b9b9101bdb0c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868762786
**[Test build #140339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140339/testReport)** for PR 32787 at commit [`b093b97`](https://github.com/apache/spark/commit/b093b97e98668cecd7f0cf52ccc830c158e0b22c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868873972
**[Test build #140339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140339/testReport)** for PR 32787 at commit [`b093b97`](https://github.com/apache/spark/commit/b093b97e98668cecd7f0cf52ccc830c158e0b22c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32787:
URL: https://github.com/apache/spark/pull/32787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r649726942
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -115,14 +115,16 @@ object FakeV2SessionCatalog extends TableCatalog {
* if `t` was a permanent table when the current view was created, it
* should still be a permanent table when resolving the current view,
* even if a temp view `t` has been created.
+ * @param outerPlans outer query plans.
*/
case class AnalysisContext(
catalogAndNamespace: Seq[String] = Nil,
nestedViewDepth: Int = 0,
maxNestedViewDepth: Int = -1,
relationCache: mutable.Map[Seq[String], LogicalPlan] = mutable.Map.empty,
referredTempViewNames: Seq[Seq[String]] = Seq.empty,
- referredTempFunctionNames: Seq[String] = Seq.empty)
+ referredTempFunctionNames: Seq[String] = Seq.empty,
+ outerPlans: Seq[LogicalPlan] = Seq.empty)
Review comment:
Could you update the comment above? https://github.com/apache/spark/pull/32787/files#diff-ed19f376a63eba52eea59ca71f3355d4495fad4fad4db9a3324aade0d4986a47R97-R99
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala
##########
@@ -177,4 +178,61 @@ class ResolveSubquerySuite extends AnalysisTest {
condition = Some(sum('a) === sum('c)))
assertAnalysisError(plan, Seq("Invalid expressions: [sum(a), sum(c)]"))
}
+
+ test("SPARK-35618: lateral join with star expansion") {
Review comment:
Could you add tests for case-sensitivities?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -115,14 +115,16 @@ object FakeV2SessionCatalog extends TableCatalog {
* if `t` was a permanent table when the current view was created, it
* should still be a permanent table when resolving the current view,
* even if a temp view `t` has been created.
+ * @param outerPlans outer query plans.
Review comment:
nit: `outer` -> `Outer`. Also, could you describe more about what's this param used for? (just like the other params)
##########
File path: sql/core/src/test/resources/sql-tests/inputs/join-lateral.sql
##########
@@ -12,9 +12,8 @@ SELECT * FROM t1, LATERAL (SELECT t1.c1 + t2.c1 FROM t2);
-- lateral join with star expansion
SELECT * FROM t1, LATERAL (SELECT *);
SELECT * FROM t1, LATERAL (SELECT * FROM t2);
--- TODO(SPARK-35618): resolve star expressions in subquery
--- SELECT * FROM t1, LATERAL (SELECT t1.*);
--- SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2);
+SELECT * FROM t1, LATERAL (SELECT t1.*);
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2);
Review comment:
Could you add tests for more deeply-nested cases? e.g.,
```
SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t3));
```
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala
##########
@@ -177,4 +178,61 @@ class ResolveSubquerySuite extends AnalysisTest {
condition = Some(sum('a) === sum('c)))
assertAnalysisError(plan, Seq("Invalid expressions: [sum(a), sum(c)]"))
}
+
+ test("SPARK-35618: lateral join with star expansion") {
Review comment:
Also, I think we need some tests for `spark.sql.parser.quotedRegexColumnNames`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867169394
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44749/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r661032351
##########
File path: sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out
##########
@@ -80,6 +80,46 @@ struct<c1:int,c2:int,c1:int,c2:int>
1 2 0 3
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1
+1 2 1 2
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1 0 2
+0 1 0 1 0 3
+1 2 1 2 0 2
+1 2 1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.* FROM t2 AS t1)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 2
+0 1 0 3
+1 2 0 2
+1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3))
Review comment:
Ah, ok. Thank you for the explanation. If there is a possibility to support it in future work, could you leave some comments there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858168705
**[Test build #139601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139601/testReport)** for PR 32787 at commit [`1704a9a`](https://github.com/apache/spark/commit/1704a9a8c54fab6b0b450c4fe32be444dc81df20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855152606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855048367
**[Test build #139356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139356/testReport)** for PR 32787 at commit [`6150cc4`](https://github.com/apache/spark/commit/6150cc4cd570ea0f8023334a2fa8149ca7b997ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858289714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870967059
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44904/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855173701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856372399
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r662013726
##########
File path: sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out
##########
@@ -80,6 +80,46 @@ struct<c1:int,c2:int,c1:int,c2:int>
1 2 0 3
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1
+1 2 1 2
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 1 0 2
+0 1 0 1 0 3
+1 2 1 2 0 2
+1 2 1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.* FROM t2 AS t1)
+-- !query schema
+struct<c1:int,c2:int,c1:int,c2:int>
+-- !query output
+0 1 0 2
+0 1 0 3
+1 2 0 2
+1 2 0 3
+
+
+-- !query
+SELECT * FROM t1, LATERAL (SELECT t1.*, t2.* FROM t2, LATERAL (SELECT t1.*, t2.*, t3.* FROM t2 AS t3))
Review comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870987961
Looks fine otherwise except for the @cloud-fan comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870985363
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44904/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r660349216
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -807,4 +807,48 @@ class AnalysisErrorSuite extends AnalysisTest {
// UnresolvedHint be removed by batch `Remove Unresolved Hints`
assertAnalysisSuccess(plan, true)
}
+
+ test("SPARK-35618: Resolve star expressions in subqueries") {
+ val a = AttributeReference("a", IntegerType)()
+ val b = AttributeReference("b", IntegerType)()
+ val t0 = OneRowRelation()
+ val t1 = LocalRelation(a, b).as("t1")
+
+ // t1.* in the subquery should be resolved into outer(t1.a) and outer(t1.b).
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(star("t1"))).as("sub") :: Nil, t1),
+ "Scalar subquery must return only one column, but got 2" :: Nil)
+
+ // array(t1.*) in the subquery should be resolved into array(outer(t1.a), outer(t1.b))
+ val array = CreateArray(Seq(star("t1")))
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(array)).as("sub") :: Nil, t1),
+ "Expressions referencing the outer query are not supported" :: Nil)
Review comment:
hmm, I think we do support star in `CreateArray`? https://github.com/apache/spark/pull/32787/files#diff-c00a384380a1462ecad9fca0fed853c47d6894867420f7daf94b1c56ae444e4bR226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32787:
URL: https://github.com/apache/spark/pull/32787#discussion_r662017168
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -807,4 +807,48 @@ class AnalysisErrorSuite extends AnalysisTest {
// UnresolvedHint be removed by batch `Remove Unresolved Hints`
assertAnalysisSuccess(plan, true)
}
+
+ test("SPARK-35618: Resolve star expressions in subqueries") {
+ val a = AttributeReference("a", IntegerType)()
+ val b = AttributeReference("b", IntegerType)()
+ val t0 = OneRowRelation()
+ val t1 = LocalRelation(a, b).as("t1")
+
+ // t1.* in the subquery should be resolved into outer(t1.a) and outer(t1.b).
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(star("t1"))).as("sub") :: Nil, t1),
+ "Scalar subquery must return only one column, but got 2" :: Nil)
+
+ // array(t1.*) in the subquery should be resolved into array(outer(t1.a), outer(t1.b))
+ val array = CreateArray(Seq(star("t1")))
+ assertAnalysisError(
+ Project(ScalarSubquery(t0.select(array)).as("sub") :: Nil, t1),
+ "Expressions referencing the outer query are not supported" :: Nil)
Review comment:
Yes star in `CreateArray` is valid. I truncated this error message but the complete one is `"Expressions referencing the outer query are not supported outside of WHERE/HAVING clauses"`. Will update the message to be clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-855131699
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872057518
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-870825377
**[Test build #140381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140381/testReport)** for PR 32787 at commit [`fe5f89a`](https://github.com/apache/spark/commit/fe5f89a323f948f152ace02418fb5866442a31ba).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-858168705
**[Test build #139601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139601/testReport)** for PR 32787 at commit [`1704a9a`](https://github.com/apache/spark/commit/1704a9a8c54fab6b0b450c4fe32be444dc81df20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-872091206
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-868821648
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-871969800
**[Test build #140491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140491/testReport)** for PR 32787 at commit [`0346d3b`](https://github.com/apache/spark/commit/0346d3b7c0cbbe3edf88a1451da9bf53726a458d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-856429384
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139437/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867140693
**[Test build #140221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140221/testReport)** for PR 32787 at commit [`8f1f42a`](https://github.com/apache/spark/commit/8f1f42a134901524872ada157c82215313d03e7d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32787:
URL: https://github.com/apache/spark/pull/32787#issuecomment-867176608
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44749/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org