You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/26 22:18:16 UTC
[GitHub] [spark] allisonwang-db opened a new pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
allisonwang-db opened a new pull request #33527:
URL: https://github.com/apache/spark/pull/33527
This PR cherry picks https://github.com/apache/spark/pull/33235 to branch-3.2 to fix test failures introduced by https://github.com/apache/spark/pull/33284.
### What changes were proposed in this pull request?
This PR allows the `Project` node to host outer references in scalar subqueries when `decorrelateInnerQuery` is enabled. It is already supported by the new decorrelation framework and the `RewriteCorrelatedScalarSubquery` rule.
Note currently by default all correlated subqueries will be decorrelated, which is not necessarily the most optimal approach. Consider `SELECT (SELECT c1) FROM t`. This should be optimized as `SELECT c1 FROM t` instead of rewriting it as a left outer join. This will be done in a separate PR to optimize correlated scalar/lateral subqueries with OneRowRelation.
### Why are the changes needed?
To allow more types of correlated scalar subqueries.
### Does this PR introduce _any_ user-facing change?
Yes. This PR allows outer query column references in the SELECT cluase of a correlated scalar subquery. For example:
```sql
SELECT (SELECT c1) FROM t;
```
Before this change:
```
org.apache.spark.sql.AnalysisException: Expressions referencing the outer query are not supported
outside of WHERE/HAVING clauses
```
After this change:
```
+------------------+
|scalarsubquery(c1)|
+------------------+
|0 |
|1 |
+------------------+
```
### How was this patch tested?
Added unit tests and SQL tests.
(cherry picked from commit ca348e50a4edbd857ec86e4e9693fa4bcbab54b7)
Signed-off-by: allisonwang-db <al...@databricks.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887176450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141664/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887792143
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46240/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887918953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887066192
**[Test build #141664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141664/testReport)** for PR 33527 at commit [`fb58c1c`](https://github.com/apache/spark/commit/fb58c1cbdf103e3fe63d5d7acf5a70c2bd91b23d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-888009030
thanks, merging to 3.2!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887164127
**[Test build #141664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141664/testReport)** for PR 33527 at commit [`fb58c1c`](https://github.com/apache/spark/commit/fb58c1cbdf103e3fe63d5d7acf5a70c2bd91b23d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33527:
URL: https://github.com/apache/spark/pull/33527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887176450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141664/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887731947
**[Test build #141727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141727/testReport)** for PR 33527 at commit [`29b068c`](https://github.com/apache/spark/commit/29b068c66caa9fc111701e72fa727072227bf2be).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887115662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887066192
**[Test build #141664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141664/testReport)** for PR 33527 at commit [`fb58c1c`](https://github.com/apache/spark/commit/fb58c1cbdf103e3fe63d5d7acf5a70c2bd91b23d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887809119
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46240/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887918953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887115662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887731947
**[Test build #141727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141727/testReport)** for PR 33527 at commit [`29b068c`](https://github.com/apache/spark/commit/29b068c66caa9fc111701e72fa727072227bf2be).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887900697
**[Test build #141727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141727/testReport)** for PR 33527 at commit [`29b068c`](https://github.com/apache/spark/commit/29b068c66caa9fc111701e72fa727072227bf2be).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887090448
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887169657
cc @cloud-fan (The test failures seem to be related to SPARK-35985?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887809119
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46240/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33527: [SPARK-36028][SQL][3.2] Allow Project to host outer references in scalar subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33527:
URL: https://github.com/apache/spark/pull/33527#issuecomment-887103153
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org