You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/26 20:19:34 UTC
[GitHub] [spark] imback82 opened a new pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
imback82 opened a new pull request #31352:
URL: https://github.com/apache/spark/pull/31352
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
The following fails:
```scala
sql("create temporary view ta(a, b) as select 1, 2")
sql("create temporary view tc(c, d) as select 1, 2")
sql("select a, (select sum(d) from tc where a = c) sum_d from ta group by 1, 2").show
```
with
```
This method should not be called in the analyzer
java.lang.RuntimeException: This method should not be called in the analyzer
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.assertNotAnalysisRule(AnalysisHelper.scala:159)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.assertNotAnalysisRule$(AnalysisHelper.scala:155)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.assertNotAnalysisRule(LogicalPlan.scala:29)
```
because `EliminateView` which uses `transformUp` is called by `View.doCanonicalize` during analysis phase (`CheckAnalysis`). This PR proposes to fix it by wrapping `transformUp` with `AnalysisHelper.allowInvokingTransformsInAnalyzer`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To fix a regression from Spark 3.0.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, now the above query works fine.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added a new test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767838747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39104/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770271596
**[Test build #134678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134678/testReport)** for PR 31352 at commit [`76772e7`](https://github.com/apache/spark/commit/76772e720c693d26f6b6befdbabd8652d9180acb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768072620
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39121/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767975461
@cloud-fan Is this an issue if it fails only under tests (assertNotAnalysisRule() check), or no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768190378
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134536/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768023838
> Let's remove the regression mentioned in PR description. Seems like it's not a regression.
In Spark 3.1.1-RC, if we start `spark-shell` with the `SPARK_TESTING=true` env variable, we see the exception whereas we don't see it in Spark 3.0. Since the exception is thrown only for "testing" mode, this would not be considered a regression?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 edited a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 edited a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767856221
Thanks @dongjoon-hyun for checking. Let me double-check and get back to you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767838747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39104/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770302162
**[Test build #134678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134678/testReport)** for PR 31352 at commit [`76772e7`](https://github.com/apache/spark/commit/76772e720c693d26f6b6befdbabd8652d9180acb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767945206
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134519/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768060093
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39121/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770304990
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134678/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768018643
Let's remove the regression mentioned in PR description. Seems like it's not a regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770287681
> Hi @imback82 , can you re-open the PR to add the tests? thanks!
Updated, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770275806
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39265/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770271596
**[Test build #134678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134678/testReport)** for PR 31352 at commit [`76772e7`](https://github.com/apache/spark/commit/76772e720c693d26f6b6befdbabd8652d9180acb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768510068
@imback82 thanks for catching this issue! Since it only affects tests, I don't believe it's a real bug.
I think calling a rule in `doCanonicalize` is tricky, so I opened a PR to clean up this: https://github.com/apache/spark/pull/31368 , please take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31352:
URL: https://github.com/apache/spark/pull/31352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768045693
**[Test build #134536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134536/testReport)** for PR 31352 at commit [`7016fa5`](https://github.com/apache/spark/commit/7016fa50a4a61d4869f5c7770089f9096fb256b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768183884
**[Test build #134536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134536/testReport)** for PR 31352 at commit [`7016fa5`](https://github.com/apache/spark/commit/7016fa50a4a61d4869f5c7770089f9096fb256b8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767807381
**[Test build #134519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134519/testReport)** for PR 31352 at commit [`651a2ec`](https://github.com/apache/spark/commit/651a2ecb7d46f376aaab776593ec245b79b9904b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767856221
Thanks @dongjoon-hyun for checking. Let me double-check and get back to you.l
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767945206
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134519/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767939867
Ah, it's failing only under tests:
```scala
protected def assertNotAnalysisRule(): Unit = {
if (Utils.isTesting &&
AnalysisHelper.inAnalyzer.get > 0 &&
AnalysisHelper.resolveOperatorDepth.get == 0) {
throw new RuntimeException("This method should not be called in the analyzer")
}
}
```
`Utils.isTesting` is true if `SPARK_TESTING` env variable is set.
Is this OK?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31352:
URL: https://github.com/apache/spark/pull/31352#discussion_r565018271
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
##########
@@ -53,38 +53,42 @@ import org.apache.spark.sql.catalyst.rules.Rule
* completely resolved during the batch of Resolution.
*/
object EliminateView extends Rule[LogicalPlan] with CastSupport {
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- // The child has the different output attributes with the View operator. Adds a Project over
- // the child of the view.
- case v @ View(desc, _, output, child) if child.resolved && !v.sameOutput(child) =>
- val resolver = conf.resolver
- val queryColumnNames = desc.viewQueryColumnNames
- val queryOutput = if (queryColumnNames.nonEmpty) {
- // Find the attribute that has the expected attribute name from an attribute list, the names
- // are compared using conf.resolver.
- // `CheckAnalysis` already guarantees the expected attribute can be found for sure.
- desc.viewQueryColumnNames.map { colName =>
- child.output.find(attr => resolver(attr.name, colName)).get
- }
- } else {
- // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
- // output is the same with the view output.
- child.output
- }
- // Map the attributes in the query output to the attributes in the view output by index.
- val newOutput = output.zip(queryOutput).map {
- case (attr, originAttr) if !attr.semanticEquals(originAttr) =>
- // `CheckAnalysis` already guarantees that the cast is a up-cast for sure.
- Alias(cast(originAttr, attr.dataType), attr.name)(exprId = attr.exprId,
- qualifier = attr.qualifier, explicitMetadata = Some(attr.metadata))
- case (_, originAttr) => originAttr
- }
- Project(newOutput, child)
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ AnalysisHelper.allowInvokingTransformsInAnalyzer {
+ plan transformUp {
Review comment:
or I would just simply do:
```scala
AnalysisHelper.allowInvokingTransformsInAnalyzer { plan transformUp {
...
}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770285131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768529453
Thank you, @imback82 , @HyukjinKwon , and @cloud-fan .
Ya, @cloud-fan 's new PR might be a better approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31352:
URL: https://github.com/apache/spark/pull/31352#discussion_r564807595
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
##########
@@ -53,38 +53,42 @@ import org.apache.spark.sql.catalyst.rules.Rule
* completely resolved during the batch of Resolution.
*/
object EliminateView extends Rule[LogicalPlan] with CastSupport {
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- // The child has the different output attributes with the View operator. Adds a Project over
- // the child of the view.
- case v @ View(desc, _, output, child) if child.resolved && !v.sameOutput(child) =>
- val resolver = conf.resolver
- val queryColumnNames = desc.viewQueryColumnNames
- val queryOutput = if (queryColumnNames.nonEmpty) {
- // Find the attribute that has the expected attribute name from an attribute list, the names
- // are compared using conf.resolver.
- // `CheckAnalysis` already guarantees the expected attribute can be found for sure.
- desc.viewQueryColumnNames.map { colName =>
- child.output.find(attr => resolver(attr.name, colName)).get
- }
- } else {
- // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
- // output is the same with the view output.
- child.output
- }
- // Map the attributes in the query output to the attributes in the view output by index.
- val newOutput = output.zip(queryOutput).map {
- case (attr, originAttr) if !attr.semanticEquals(originAttr) =>
- // `CheckAnalysis` already guarantees that the cast is a up-cast for sure.
- Alias(cast(originAttr, attr.dataType), attr.name)(exprId = attr.exprId,
- qualifier = attr.qualifier, explicitMetadata = Some(attr.metadata))
- case (_, originAttr) => originAttr
- }
- Project(newOutput, child)
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ AnalysisHelper.allowInvokingTransformsInAnalyzer {
+ plan transformUp {
Review comment:
There is no change below this line. If you don't like the diff, I can change the PR to something like the following:
```scala
override def apply(plan: LogicalPlan): LogicalPlan = {
AnalysisHelper.allowInvokingTransformsInAnalyzer {
applyInternal(plan)
}
}
private def applyInternal(plan: LogicalPlan): LogicalPlan = plan transformUp {
```
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
##########
@@ -512,6 +512,18 @@ class SubquerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
Row(1, null) :: Row(2, 6.0) :: Row(3, 2.0) :: Row(null, null) :: Row(6, null) :: Nil)
}
+ test("SPARK-34252: correlated scalar subquery with view in aggregate") {
+ withTable("tr") {
+ withView("vr") {
+ r.write.saveAsTable("tr")
+ sql("create view vr as select * from tr")
+ checkAnswer(
+ sql("select a, (select sum(d) from vr where a = c) sum_d from l l1 group by 1, 2"),
Review comment:
Note that the existing test with the same query worked fine because `r` is a dataframe temp view, which doesn't have the `View` node.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768544554
Thanks all! I'll close this PR and check the other one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770279175
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39265/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768093676
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39121/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 closed pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 closed pull request #31352:
URL: https://github.com/apache/spark/pull/31352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768093676
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39121/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767933687
**[Test build #134519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134519/testReport)** for PR 31352 at commit [`651a2ec`](https://github.com/apache/spark/commit/651a2ecb7d46f376aaab776593ec245b79b9904b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31352: [SPARK-34269][SQL][TESTS][FOLLOWUP] Test a subquery with view in aggregate's grouping expression
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-770285131
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39265/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31352:
URL: https://github.com/apache/spark/pull/31352#discussion_r565027977
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
##########
@@ -53,38 +53,42 @@ import org.apache.spark.sql.catalyst.rules.Rule
* completely resolved during the batch of Resolution.
*/
object EliminateView extends Rule[LogicalPlan] with CastSupport {
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- // The child has the different output attributes with the View operator. Adds a Project over
- // the child of the view.
- case v @ View(desc, _, output, child) if child.resolved && !v.sameOutput(child) =>
- val resolver = conf.resolver
- val queryColumnNames = desc.viewQueryColumnNames
- val queryOutput = if (queryColumnNames.nonEmpty) {
- // Find the attribute that has the expected attribute name from an attribute list, the names
- // are compared using conf.resolver.
- // `CheckAnalysis` already guarantees the expected attribute can be found for sure.
- desc.viewQueryColumnNames.map { colName =>
- child.output.find(attr => resolver(attr.name, colName)).get
- }
- } else {
- // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
- // output is the same with the view output.
- child.output
- }
- // Map the attributes in the query output to the attributes in the view output by index.
- val newOutput = output.zip(queryOutput).map {
- case (attr, originAttr) if !attr.semanticEquals(originAttr) =>
- // `CheckAnalysis` already guarantees that the cast is a up-cast for sure.
- Alias(cast(originAttr, attr.dataType), attr.name)(exprId = attr.exprId,
- qualifier = attr.qualifier, explicitMetadata = Some(attr.metadata))
- case (_, originAttr) => originAttr
- }
- Project(newOutput, child)
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ AnalysisHelper.allowInvokingTransformsInAnalyzer {
+ plan transformUp {
Review comment:
Thanks @HyukjinKwon for the suggestion! Since `allowInvokingTransformsInAnalyzer` introduces one more indentation, I had to move it to align with `override`, which I am not sure is the style we want to follow. Please let me know!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-769614946
Hi @imback82 , can you re-open the PR to add the tests? thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768045693
**[Test build #134536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134536/testReport)** for PR 31352 at commit [`7016fa5`](https://github.com/apache/spark/commit/7016fa50a4a61d4869f5c7770089f9096fb256b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767900491
Thanks. Maybe, do I need to use some configuration?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-767807381
**[Test build #134519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134519/testReport)** for PR 31352 at commit [`651a2ec`](https://github.com/apache/spark/commit/651a2ecb7d46f376aaab776593ec245b79b9904b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31352: [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31352:
URL: https://github.com/apache/spark/pull/31352#issuecomment-768190378
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134536/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org