You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/22 19:06:13 UTC
[GitHub] [spark] imback82 opened a new pull request #31933: [W[SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
imback82 opened a new pull request #31933:
URL: https://github.com/apache/spark/pull/31933
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR proposes to remove re-analyzing the already analyzed plan for `CreateViewCommand` as discussed https://github.com/apache/spark/pull/31273/files#r581592786.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
No need to analyze the plan if it's already analyzed.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing tests should cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r602392219
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -96,11 +96,6 @@ case class CreateViewCommand(
}
override def run(sparkSession: SparkSession): Seq[Row] = {
- // If the plan cannot be analyzed, throw an exception and don't proceed.
- val qe = sparkSession.sessionState.executePlan(child)
- qe.assertAnalyzed()
- val analyzedPlan = qe.analyzed
Review comment:
OK, I will give it a shot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804508162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806332239
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806220275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806288874
**[Test build #136486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136486/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806162939
**[Test build #136471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136471/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804602370
**[Test build #136379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136379/testReport)** for PR 31933 at commit [`6fdd9e0`](https://github.com/apache/spark/commit/6fdd9e0edc924679420d3c64472b72e83bb6006f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599266970
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
##########
@@ -28,6 +28,7 @@ trait Command extends LogicalPlan {
override def output: Seq[Attribute] = Seq.empty
override def producedAttributes: AttributeSet = outputSet
override def children: Seq[LogicalPlan] = Seq.empty
+ def plansToCheckAnalysis: Seq[LogicalPlan] = Seq.empty
Review comment:
can we reuse `innerChildren`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
##########
@@ -28,6 +28,7 @@ trait Command extends LogicalPlan {
override def output: Seq[Attribute] = Seq.empty
override def producedAttributes: AttributeSet = outputSet
override def children: Seq[LogicalPlan] = Seq.empty
+ def plansToCheckAnalysis: Seq[LogicalPlan] = Seq.empty
Review comment:
can we reuse `innerChildren` instead of adding this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599257472
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -167,6 +167,9 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
case _: ShowTableExtended =>
throw new AnalysisException("SHOW TABLE EXTENDED is not supported for v2 tables.")
+ case c: Command =>
+ c.plansToCheckAnalysis.foreach(checkAnalysis)
Review comment:
This seems hacky? But we cannot make the analyzed plan as `children` for `CreateViewCommaned`. The reason is that the `View` will be optimized away (in the optimizer), and the verification that checks if a permanent view references temp views will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804485188
**[Test build #136368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136368/testReport)** for PR 31933 at commit [`5829e48`](https://github.com/apache/spark/commit/5829e484f8538bd1529ac4386fff1848634c2997).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806206407
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804674725
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806312865
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804485188
**[Test build #136368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136368/testReport)** for PR 31933 at commit [`5829e48`](https://github.com/apache/spark/commit/5829e484f8538bd1529ac4386fff1848634c2997).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804674725
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804508164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806163848
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136471/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804330252
**[Test build #136359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136359/testReport)** for PR 31933 at commit [`ff7fabb`](https://github.com/apache/spark/commit/ff7fabb55812cfdb5cfd596cc100cef6391f1381).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806332856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804347165
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40943/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804335559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804330252
**[Test build #136359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136359/testReport)** for PR 31933 at commit [`ff7fabb`](https://github.com/apache/spark/commit/ff7fabb55812cfdb5cfd596cc100cef6391f1381).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806342125
**[Test build #136486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136486/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-805963133
**[Test build #136471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136471/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804637310
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40963/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806163848
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136471/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806350798
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136486/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599269087
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -62,15 +62,17 @@ case class CreateViewCommand(
comment: Option[String],
properties: Map[String, String],
originalText: Option[String],
- child: LogicalPlan,
+ analyzedPlan: LogicalPlan,
Review comment:
how do we analyze it since it's not a child?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r606017149
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -96,11 +96,6 @@ case class CreateViewCommand(
}
override def run(sparkSession: SparkSession): Seq[Row] = {
- // If the plan cannot be analyzed, throw an exception and don't proceed.
- val qe = sparkSession.sessionState.executePlan(child)
- qe.assertAnalyzed()
- val analyzedPlan = qe.analyzed
Review comment:
@cloud-fan I created https://github.com/apache/spark/pull/32032 to handle this scenario. Please let me know what you think. TIA!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804505407
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40952/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804617321
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40963/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-805963133
**[Test build #136471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136471/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806393926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r600238917
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -683,6 +683,18 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
case _ => // Analysis successful!
}
}
+
+ // Check analysis on internal nodes.
+ plan match {
+ case c: Command =>
Review comment:
shall we move it to L682?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806288874
**[Test build #136486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136486/testReport)** for PR 31933 at commit [`a14b3b9`](https://github.com/apache/spark/commit/a14b3b921a778d1b4236945d64a1e62648ba1430).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806288292
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804335559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806491362
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136496/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804507175
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40952/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r602390426
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -96,11 +96,6 @@ case class CreateViewCommand(
}
override def run(sparkSession: SparkSession): Seq[Row] = {
- // If the plan cannot be analyzed, throw an exception and don't proceed.
- val qe = sparkSession.sessionState.executePlan(child)
- qe.assertAnalyzed()
- val analyzedPlan = qe.analyzed
Review comment:
It will be great to have such a mechanism to skip optimizer, useful to `CacheTableAsSelect` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806482755
**[Test build #136496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136496/testReport)** for PR 31933 at commit [`507b00c`](https://github.com/apache/spark/commit/507b00c34058cf52f430641078e44a0c1fc0be84).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class GetViewColumnByNameAndOrdinal(`
* `case class MultiplyYMInterval(`
* `case class LocalLimitExec(limit: Int, child: SparkPlan) extends BaseLimitExec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806379672
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806383471
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-805086360
**[Test build #136418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136418/testReport)** for PR 31933 at commit [`e04fbdd`](https://github.com/apache/spark/commit/e04fbdda9e088173c956e24d4214f17bca640e29).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806332856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806166879
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806352100
**[Test build #136496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136496/testReport)** for PR 31933 at commit [`507b00c`](https://github.com/apache/spark/commit/507b00c34058cf52f430641078e44a0c1fc0be84).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804347519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40943/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599268594
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
##########
@@ -94,14 +94,19 @@ case class CacheTableAsSelectExec(
override lazy val relationName: String = tempViewName
override lazy val planToCache: LogicalPlan = {
+ // If the plan cannot be analyzed, throw an exception and don't proceed.
+ val qe = sparkSession.sessionState.executePlan(query)
+ qe.assertAnalyzed()
+ val analyzedPlan = qe.analyzed
Review comment:
The current code looks fine. I think `CacheTableAsSelectExec` is the only exception that it has a `query` which is not a simple table relation but we want to skip optimizing it. Let's document this clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r602383484
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -96,11 +96,6 @@ case class CreateViewCommand(
}
override def run(sparkSession: SparkSession): Seq[Row] = {
- // If the plan cannot be analyzed, throw an exception and don't proceed.
- val qe = sparkSession.sessionState.executePlan(child)
- qe.assertAnalyzed()
- val analyzedPlan = qe.analyzed
Review comment:
Good catch. One issue with "make CreateViewCommand.analyzedPlan a real child" is that it will become an optimized plan, which will affect the caching. I can try to make certain command's children to skip optimizer. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804645449
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40963/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804335201
**[Test build #136359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136359/testReport)** for PR 31933 at commit [`ff7fabb`](https://github.com/apache/spark/commit/ff7fabb55812cfdb5cfd596cc100cef6391f1381).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599778249
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
##########
@@ -94,14 +94,19 @@ case class CacheTableAsSelectExec(
override lazy val relationName: String = tempViewName
override lazy val planToCache: LogicalPlan = {
+ // If the plan cannot be analyzed, throw an exception and don't proceed.
+ val qe = sparkSession.sessionState.executePlan(query)
+ qe.assertAnalyzed()
+ val analyzedPlan = qe.analyzed
Review comment:
thanks, updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -62,15 +62,17 @@ case class CreateViewCommand(
comment: Option[String],
properties: Map[String, String],
originalText: Option[String],
- child: LogicalPlan,
+ analyzedPlan: LogicalPlan,
Review comment:
The analyzed plan is passed here. I can keep the name `child` if that's clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599008218
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
##########
@@ -94,14 +94,19 @@ case class CacheTableAsSelectExec(
override lazy val relationName: String = tempViewName
override lazy val planToCache: LogicalPlan = {
+ // If the plan cannot be analyzed, throw an exception and don't proceed.
+ val qe = sparkSession.sessionState.executePlan(query)
+ qe.assertAnalyzed()
+ val analyzedPlan = qe.analyzed
Review comment:
@cloud-fan If we make `query` as `child` of `CacheTableAsSelectExec`, it will be an optimized plan instead of an analyzed plan, which can be problematic when looking up the cached plan. One way to work around is to move matching `CacheTableAsSelect` back to `ResolveSessionCatalog`... Any thought?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r599256804
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -62,15 +62,17 @@ case class CreateViewCommand(
comment: Option[String],
properties: Map[String, String],
originalText: Option[String],
- child: LogicalPlan,
+ analyzedPlan: LogicalPlan,
allowExisting: Boolean,
replace: Boolean,
viewType: ViewType)
extends RunnableCommand {
import ViewHelper._
- override def innerChildren: Seq[QueryPlan[_]] = Seq(child)
+ override def plansToCheckAnalysis: Seq[LogicalPlan] = Seq(analyzedPlan)
Review comment:
We need to run checkAnalysis on the analyzed plan, otherwise, for the following:
```
sql("CREATE TABLE view_base_table (key int, data varchar(20)) USING PARQUET")
sql("CREATE VIEW key_dependent_view AS SELECT * FROM view_base_table GROUP BY key")
```
, view creation works fine, whereas it should have failed with:
```
org.apache.spark.sql.AnalysisException
expression 'spark_catalog.default.view_base_table.data' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31933:
URL: https://github.com/apache/spark/pull/31933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806409795
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r602063863
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##########
@@ -96,11 +96,6 @@ case class CreateViewCommand(
}
override def run(sparkSession: SparkSession): Seq[Row] = {
- // If the plan cannot be analyzed, throw an exception and don't proceed.
- val qe = sparkSession.sessionState.executePlan(child)
- qe.assertAnalyzed()
- val analyzedPlan = qe.analyzed
Review comment:
Actually this is a regression. `CreateViewCommand` can be created from `CreateViewStatement` in `ResolveSessionCatalog`. However, `ResolveSessionCatalog` is inside `extendedResolutionRules`, which means the `analyzedPlan` doesn't go through `postHocResolutionRules`.
I think we need to make `CreateViewCommand.analyzedPlan` a real child, so that it can go through all the analyzer rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806393926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804645449
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40963/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31933:
URL: https://github.com/apache/spark/pull/31933#discussion_r602047916
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
##########
@@ -94,14 +94,21 @@ case class CacheTableAsSelectExec(
override lazy val relationName: String = tempViewName
override lazy val planToCache: LogicalPlan = {
+ // CacheTableAsSelectExec.query is not resolved yet (e.g., not a child of CacheTableAsSelect)
+ // in order to skip optimizing it; note that we need to pass an analyzed plan to
+ // CreateViewCommand for the cache to work correctly. Thus, the query is analyzed below.
Review comment:
Can we further clean this up by creating `CreateViewStatement` instead of `CreateViewCommand` below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804602370
**[Test build #136379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136379/testReport)** for PR 31933 at commit [`6fdd9e0`](https://github.com/apache/spark/commit/6fdd9e0edc924679420d3c64472b72e83bb6006f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806350798
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136486/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804674007
**[Test build #136379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136379/testReport)** for PR 31933 at commit [`6fdd9e0`](https://github.com/apache/spark/commit/6fdd9e0edc924679420d3c64472b72e83bb6006f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806491362
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136496/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31933: [WIP][SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804347519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40943/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806352100
**[Test build #136496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136496/testReport)** for PR 31933 at commit [`507b00c`](https://github.com/apache/spark/commit/507b00c34058cf52f430641078e44a0c1fc0be84).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-806220275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-804503139
**[Test build #136368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136368/testReport)** for PR 31933 at commit [`5829e48`](https://github.com/apache/spark/commit/5829e484f8538bd1529ac4386fff1848634c2997).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31933: [SPARK-34701][SQL] Remove analyzing temp view again in CreateViewCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31933:
URL: https://github.com/apache/spark/pull/31933#issuecomment-808001885
I've reverted it due to the regression mentioned in https://github.com/apache/spark/pull/31933#discussion_r602063863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org