You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/27 19:01:13 UTC
[GitHub] [spark] cloud-fan opened a new pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
cloud-fan opened a new pull request #31368:
URL: https://github.com/apache/spark/pull/31368
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
The currently SQL (temp or permanent) view resolution is done in 2 steps:
1. In `SessionCatalog`, we get the view metadata, parse the view SQL string, and wrap it with `View`.
2. At the beginning of the optimizer, we run `EliminateView`, which drops the wrapper `View`, and apply some special logic to match the view schema.
Step 2 is tricky, as we need to retain the output attr expr id, while we need to add an extra `Project` to add cast and alias. This PR simplifies the view solution by building a completed (with cast and alias added) in `SessionCatalog`, so that we only have 1 step.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Code simplification. It also fixes issues like https://github.com/apache/spark/pull/31352
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
existing tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768896792
**[Test build #134605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134605/testReport)** for PR 31368 at commit [`dfc9d9d`](https://github.com/apache/spark/commit/dfc9d9d0921c81fde222107d3b4b35c5bb180c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565560459
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -422,41 +422,6 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
case write: V2WriteCommand if write.resolved =>
write.query.schema.foreach(f => TypeUtils.failWithIntervalType(f.dataType))
- // If the view output doesn't have the same number of columns neither with the child
- // output, nor with the query column names, throw an AnalysisException.
- // If the view's child output can't up cast to the view output,
- // throw an AnalysisException, too.
- case v @ View(desc, _, output, child) if child.resolved && !v.sameOutput(child) =>
Review comment:
This is not needed anymore, because
1. `View.output` now directly comes from `child.output`
2. The `UpCast` is added to the plan, and will go through its own error reporting branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566575428
##########
File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out
##########
@@ -796,14 +796,16 @@ IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few comman
: +- Filter (dept_id#x = outer(dept_id#x))
: +- SubqueryAlias dept
: +- View (`DEPT`, [dept_id#x,dept_name#x,state#x])
-: +- Project [dept_id#x, dept_name#x, state#x]
-: +- SubqueryAlias DEPT
-: +- LocalRelation [dept_id#x, dept_name#x, state#x]
+: +- Project [cast(dept_id#x as int) AS dept_id#x, cast(dept_name#x as string) AS dept_name#x, cast(state#x as string) AS state#x]
Review comment:
It's better to delay the cast adding (after the parsed view plan is resolved), so that we can skip adding cast for views that have no schema changing. But I can't find an easy way to do it and this is really not a big deal (optimizer willl remove redundant casts), so I go with the simple approach for maintainability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768526213
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39151/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768750394
Interesting. This seems to overlap with [SPARK-34108](https://issues.apache.org/jira/browse/SPARK-34108) but it appears that it doesn't solve the issue in the JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768813235
@cloud-fan it's working now - thanks! I'll close the JIRA as duplicate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565881107
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
+ } else {
+ // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
+ // output is the same with the view output.
+ parsedPlan
Review comment:
ah but I should still add cast, to match the behavior before this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769104935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39204/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565563247
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -625,7 +625,8 @@ case class DescribeTableCommand(
throw new AnalysisException(
s"DESC PARTITION is not allowed on a temporary view: ${table.identifier}")
}
- describeSchema(catalog.lookupRelation(table).schema, result, header = false)
+ val schema = catalog.getTempViewOrPermanentTableMetadata(table).schema
Review comment:
the view plan can be unresolved (with cast and alias added), we should use the recorded view schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769356754
**[Test build #134621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134621/testReport)** for PR 31368 at commit [`2587e53`](https://github.com/apache/spark/commit/2587e538e6f76e214a9d26685526d43f97882395).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768810389
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769048526
**[Test build #134616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134616/testReport)** for PR 31368 at commit [`a467ed6`](https://github.com/apache/spark/commit/a467ed6d68d36acdf4e3cbbfbc4c862d4842327d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565880329
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
+ } else {
+ // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
+ // output is the same with the view output.
+ parsedPlan
Review comment:
Before Spark 2.2.0, we generate SQL from logical plan, and the logical plan already has extra `Project` to add alias, see https://github.com/apache/spark/blob/branch-2.1/sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala#L214
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768998339
**[Test build #134616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134616/testReport)** for PR 31368 at commit [`a467ed6`](https://github.com/apache/spark/commit/a467ed6d68d36acdf4e3cbbfbc4c862d4842327d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768532277
**[Test build #134565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134565/testReport)** for PR 31368 at commit [`640a36b`](https://github.com/apache/spark/commit/640a36be45be30550aa09fc8eee7b7f182ec9987).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565816065
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
##########
@@ -17,74 +17,21 @@
package org.apache.spark.sql.catalyst.analysis
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, View}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, View}
import org.apache.spark.sql.catalyst.rules.Rule
/**
* This file defines view types and analysis rules related to views.
*/
/**
- * This rule has two goals:
- *
- * 1. Removes [[View]] operators from the plan. The operator is respected till the end of analysis
- * stage because we want to see which part of an analyzed logical plan is generated from a view.
- *
- * 2. Make sure that a view's child plan produces the view's output attributes. We try to wrap the
- * child by:
- * 1. Generate the `queryOutput` by:
- * 1.1. If the query column names are defined, map the column names to attributes in the child
- * output by name(This is mostly for handling view queries like SELECT * FROM ..., the
- * schema of the referenced table/view may change after the view has been created, so we
- * have to save the output of the query to `viewQueryColumnNames`, and restore them during
- * view resolution, in this way, we are able to get the correct view column ordering and
- * omit the extra columns that we don't require);
- * 1.2. Else set the child output attributes to `queryOutput`.
- * 2. Map the `queryOutput` to view output by index, if the corresponding attributes don't match,
- * try to up cast and alias the attribute in `queryOutput` to the attribute in the view output.
- * 3. Add a Project over the child, with the new output generated by the previous steps.
- *
- * Once reaches this rule, it means `CheckAnalysis` did necessary checks on number of columns
- * between the view output and the child output or the query column names. `CheckAnalysis` also
- * checked the cast from the view's child to the Project is up-cast.
- *
- * This should be only done after the batch of Resolution, because the view attributes are not
- * completely resolved during the batch of Resolution.
+ * This rule removes [[View]] operators from the plan. The operator is respected till the end of
+ * analysis stage because we want to see which part of an analyzed logical plan is generated from a
+ * view.
*/
object EliminateView extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
Review comment:
seems doesn't matter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769370153
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768898106
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134588/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769104935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39204/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768581840
**[Test build #134565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134565/testReport)** for PR 31368 at commit [`640a36b`](https://github.com/apache/spark/commit/640a36be45be30550aa09fc8eee7b7f182ec9987).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768924274
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134605/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566577976
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##########
@@ -230,7 +230,7 @@ object LogicalPlanIntegrity {
// NOTE: we still need to filter resolved expressions here because the output of
// some resolved logical plans can have unresolved references,
// e.g., outer references in `ExistenceJoin`.
- p.output.filter(_.resolved).map { a => (a.exprId, a.dataType) }
+ p.output.filter(_.resolved).map { a => (a.exprId, a.dataType.asNullable) }
Review comment:
got it. thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566575842
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##########
@@ -230,7 +230,7 @@ object LogicalPlanIntegrity {
// NOTE: we still need to filter resolved expressions here because the output of
// some resolved logical plans can have unresolved references,
// e.g., outer references in `ExistenceJoin`.
- p.output.filter(_.resolved).map { a => (a.exprId, a.dataType) }
+ p.output.filter(_.resolved).map { a => (a.exprId, a.dataType.asNullable) }
Review comment:
The view tests fail without this change. It's a test only thing that we don't need to backport, so I didn't spend time putting this into a separate PR with tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768794925
**[Test build #134588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134588/testReport)** for PR 31368 at commit [`2c66b39`](https://github.com/apache/spark/commit/2c66b395166c56dc3ca1b867bf2b1af06e6a6d05).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769587192
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39227/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768507410
cc @linhongliu-db @imback82 @maropu @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769176972
**[Test build #134621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134621/testReport)** for PR 31368 at commit [`2587e53`](https://github.com/apache/spark/commit/2587e538e6f76e214a9d26685526d43f97882395).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768896792
**[Test build #134605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134605/testReport)** for PR 31368 at commit [`dfc9d9d`](https://github.com/apache/spark/commit/dfc9d9d0921c81fde222107d3b4b35c5bb180c37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769055424
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134616/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768814873
@sunchao it's still valuable to keep your PR and add tests :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566578589
##########
File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out
##########
@@ -796,14 +796,16 @@ IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few comman
: +- Filter (dept_id#x = outer(dept_id#x))
: +- SubqueryAlias dept
: +- View (`DEPT`, [dept_id#x,dept_name#x,state#x])
-: +- Project [dept_id#x, dept_name#x, state#x]
-: +- SubqueryAlias DEPT
-: +- LocalRelation [dept_id#x, dept_name#x, state#x]
+: +- Project [cast(dept_id#x as int) AS dept_id#x, cast(dept_name#x as string) AS dept_name#x, cast(state#x as string) AS state#x]
Review comment:
sounds okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768590616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768795364
@sunchao I fixed some places, can you try again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566323136
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -450,45 +450,40 @@ case class InsertIntoDir(
*
* @param desc A view description(CatalogTable) that provides necessary information to resolve the
* view.
- * @param output The output of a view operator, this is generated during planning the view, so that
* we are able to decouple the output from the underlying structure.
* @param child The logical plan of a view operator, it should be a logical plan parsed from the
* `CatalogTable.viewText`, should throw an error if the `viewText` is not defined.
*/
case class View(
desc: CatalogTable,
isTempView: Boolean,
- output: Seq[Attribute],
- child: LogicalPlan) extends LogicalPlan with MultiInstanceRelation {
-
- override def producedAttributes: AttributeSet = outputSet
-
- override lazy val resolved: Boolean = child.resolved
-
- override def children: Seq[LogicalPlan] = child :: Nil
+ child: LogicalPlan) extends UnaryNode {
- override def newInstance(): LogicalPlan = copy(output = output.map(_.newInstance()))
+ override def output: Seq[Attribute] = child.output
override def simpleString(maxFields: Int): String = {
s"View (${desc.identifier}, ${output.mkString("[", ",", "]")})"
}
- override def doCanonicalize(): LogicalPlan = {
- def sameOutput(
- outerProject: Seq[NamedExpression], innerProject: Seq[NamedExpression]): Boolean = {
- outerProject.length == innerProject.length &&
- outerProject.zip(innerProject).forall {
- case(outer, inner) => outer.name == inner.name && outer.dataType == inner.dataType
- }
- }
+ override def doCanonicalize(): LogicalPlan = child match {
+ case p: Project if p.resolved && canRemoveProject(p) => p.child.canonicalized
+ case _ => child.canonicalized
+ }
- val eliminated = EliminateView(this) match {
- case Project(viewProjectList, child @ Project(queryProjectList, _))
- if sameOutput(viewProjectList, queryProjectList) =>
- child
- case other => other
+ // When resolving a SQL view, we use an extra Project to add cast and alias to make sure the view
+ // output schema doesn't change even if the table referenced by the view is changed after view
+ // creation. We should remove this extra Project during canonicalize if it does nothing.
+ // See more details in `SessionCatalog.fromCatalogTable`.
+ private def canRemoveProject(p: Project): Boolean = {
+ p.output.length == p.child.output.length && p.projectList.zipWithIndex.forall {
Review comment:
nit: you can do `p.projectList.zip(p.child.output).forall` instead so that you don't need to reference the output by index?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##########
@@ -230,7 +230,7 @@ object LogicalPlanIntegrity {
// NOTE: we still need to filter resolved expressions here because the output of
// some resolved logical plans can have unresolved references,
// e.g., outer references in `ExistenceJoin`.
- p.output.filter(_.resolved).map { a => (a.exprId, a.dataType) }
+ p.output.filter(_.resolved).map { a => (a.exprId, a.dataType.asNullable) }
Review comment:
should we add a test for the query that failed with the previous logic?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566523700
##########
File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out
##########
@@ -796,14 +796,16 @@ IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few comman
: +- Filter (dept_id#x = outer(dept_id#x))
: +- SubqueryAlias dept
: +- View (`DEPT`, [dept_id#x,dept_name#x,state#x])
-: +- Project [dept_id#x, dept_name#x, state#x]
-: +- SubqueryAlias DEPT
-: +- LocalRelation [dept_id#x, dept_name#x, state#x]
+: +- Project [cast(dept_id#x as int) AS dept_id#x, cast(dept_name#x as string) AS dept_name#x, cast(state#x as string) AS state#x]
Review comment:
Can we add a rule at the end of `Analyzer` if the plan is resolved to check the top `Project` and reduce the`Cast` if is redundant ? The UpCast seems to avoid the table reference changed before view analysis but we can remove it after analysis.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769252154
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768998339
**[Test build #134616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134616/testReport)** for PR 31368 at commit [`a467ed6`](https://github.com/apache/spark/commit/a467ed6d68d36acdf4e3cbbfbc4c862d4842327d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768924639
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39193/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565882779
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
+ } else {
+ // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
+ // output is the same with the view output.
+ parsedPlan
Review comment:
I see. Then I think the comment here can be updated together. The original comment is about output qualification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769370153
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565879026
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -845,17 +845,33 @@ class SessionCatalog(
}
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
Review comment:
It's from https://github.com/apache/spark/pull/31368/files#diff-782f0d0b0d5fa6cf642285962eb0c831d9807e3f9ec2810f964292da89547e1aL38
I changed it a little bit to match the current context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768922008
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39193/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565849237
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -845,17 +845,33 @@ class SessionCatalog(
}
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
Review comment:
Ah, ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768924660
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39193/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768794925
**[Test build #134588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134588/testReport)** for PR 31368 at commit [`2c66b39`](https://github.com/apache/spark/commit/2c66b395166c56dc3ca1b867bf2b1af06e6a6d05).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769252154
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566575842
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##########
@@ -230,7 +230,7 @@ object LogicalPlanIntegrity {
// NOTE: we still need to filter resolved expressions here because the output of
// some resolved logical plans can have unresolved references,
// e.g., outer references in `ExistenceJoin`.
- p.output.filter(_.resolved).map { a => (a.exprId, a.dataType) }
+ p.output.filter(_.resolved).map { a => (a.exprId, a.dataType.asNullable) }
Review comment:
The view tests fail without this change. It's a test only thing (the check is skipped in production) that we don't need to backport, so I didn't spend time putting this into a separate PR with tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769022511
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39204/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768532277
**[Test build #134565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134565/testReport)** for PR 31368 at commit [`640a36b`](https://github.com/apache/spark/commit/640a36be45be30550aa09fc8eee7b7f182ec9987).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769587192
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39227/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768817191
@cloud-fan sure - I can reopen it later to include more test coverage for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565561910
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##########
@@ -230,7 +230,7 @@ object LogicalPlanIntegrity {
// NOTE: we still need to filter resolved expressions here because the output of
// some resolved logical plans can have unresolved references,
// e.g., outer references in `ExistenceJoin`.
- p.output.filter(_.resolved).map { a => (a.exprId, a.dataType) }
+ p.output.filter(_.resolved).map { a => (a.exprId, a.dataType.asNullable) }
Review comment:
@maropu We can eliminate cast for complex types that are compatible (only nullability is different), so the previous logic could fail valid queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768898106
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134588/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768812256
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769205659
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565562297
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -450,46 +450,22 @@ case class InsertIntoDir(
*
* @param desc A view description(CatalogTable) that provides necessary information to resolve the
* view.
- * @param output The output of a view operator, this is generated during planning the view, so that
* we are able to decouple the output from the underlying structure.
* @param child The logical plan of a view operator, it should be a logical plan parsed from the
* `CatalogTable.viewText`, should throw an error if the `viewText` is not defined.
*/
case class View(
desc: CatalogTable,
isTempView: Boolean,
- output: Seq[Attribute],
- child: LogicalPlan) extends LogicalPlan with MultiInstanceRelation {
-
- override def producedAttributes: AttributeSet = outputSet
-
- override lazy val resolved: Boolean = child.resolved
-
- override def children: Seq[LogicalPlan] = child :: Nil
+ child: LogicalPlan) extends UnaryNode {
- override def newInstance(): LogicalPlan = copy(output = output.map(_.newInstance()))
+ override def output: Seq[Attribute] = child.output
override def simpleString(maxFields: Int): String = {
s"View (${desc.identifier}, ${output.mkString("[", ",", "]")})"
}
- override def doCanonicalize(): LogicalPlan = {
- def sameOutput(
- outerProject: Seq[NamedExpression], innerProject: Seq[NamedExpression]): Boolean = {
- outerProject.length == innerProject.length &&
- outerProject.zip(innerProject).forall {
- case(outer, inner) => outer.name == inner.name && outer.dataType == inner.dataType
- }
- }
-
- val eliminated = EliminateView(this) match {
- case Project(viewProjectList, child @ Project(queryProjectList, _))
- if sameOutput(viewProjectList, queryProjectList) =>
- child
- case other => other
- }
- eliminated.canonicalized
- }
+ override def doCanonicalize(): LogicalPlan = child.canonicalized
Review comment:
@imback82 now the problem goes away.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566405558
##########
File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out
##########
@@ -796,14 +796,16 @@ IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few comman
: +- Filter (dept_id#x = outer(dept_id#x))
: +- SubqueryAlias dept
: +- View (`DEPT`, [dept_id#x,dept_name#x,state#x])
-: +- Project [dept_id#x, dept_name#x, state#x]
-: +- SubqueryAlias DEPT
-: +- LocalRelation [dept_id#x, dept_name#x, state#x]
+: +- Project [cast(dept_id#x as int) AS dept_id#x, cast(dept_name#x as string) AS dept_name#x, cast(state#x as string) AS state#x]
Review comment:
There are some newly added `cast`. Are they redundant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565789469
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -845,17 +845,33 @@ class SessionCatalog(
}
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
Review comment:
I think so, the comment is copied from the previous code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565735036
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -845,17 +845,33 @@ class SessionCatalog(
}
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
Review comment:
> For view queries like `SELECT * FROM t`, the schema of the referenced table/view may change after the view has been created.
We already have some tests for the case above somewhere?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
##########
@@ -17,74 +17,21 @@
package org.apache.spark.sql.catalyst.analysis
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, View}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, View}
import org.apache.spark.sql.catalyst.rules.Rule
/**
* This file defines view types and analysis rules related to views.
*/
/**
- * This rule has two goals:
- *
- * 1. Removes [[View]] operators from the plan. The operator is respected till the end of analysis
- * stage because we want to see which part of an analyzed logical plan is generated from a view.
- *
- * 2. Make sure that a view's child plan produces the view's output attributes. We try to wrap the
- * child by:
- * 1. Generate the `queryOutput` by:
- * 1.1. If the query column names are defined, map the column names to attributes in the child
- * output by name(This is mostly for handling view queries like SELECT * FROM ..., the
- * schema of the referenced table/view may change after the view has been created, so we
- * have to save the output of the query to `viewQueryColumnNames`, and restore them during
- * view resolution, in this way, we are able to get the correct view column ordering and
- * omit the extra columns that we don't require);
- * 1.2. Else set the child output attributes to `queryOutput`.
- * 2. Map the `queryOutput` to view output by index, if the corresponding attributes don't match,
- * try to up cast and alias the attribute in `queryOutput` to the attribute in the view output.
- * 3. Add a Project over the child, with the new output generated by the previous steps.
- *
- * Once reaches this rule, it means `CheckAnalysis` did necessary checks on number of columns
- * between the view output and the child output or the query column names. `CheckAnalysis` also
- * checked the cast from the view's child to the Project is up-cast.
- *
- * This should be only done after the batch of Resolution, because the view attributes are not
- * completely resolved during the batch of Resolution.
+ * This rule removes [[View]] operators from the plan. The operator is respected till the end of
+ * analysis stage because we want to see which part of an analyzed logical plan is generated from a
+ * view.
*/
object EliminateView extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
Review comment:
`transformUp` -> `transform`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r566510233
##########
File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out
##########
@@ -796,14 +796,16 @@ IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few comman
: +- Filter (dept_id#x = outer(dept_id#x))
: +- SubqueryAlias dept
: +- View (`DEPT`, [dept_id#x,dept_name#x,state#x])
-: +- Project [dept_id#x, dept_name#x, state#x]
-: +- SubqueryAlias DEPT
-: +- LocalRelation [dept_id#x, dept_name#x, state#x]
+: +- Project [cast(dept_id#x as int) AS dept_id#x, cast(dept_name#x as string) AS dept_name#x, cast(state#x as string) AS state#x]
Review comment:
Redundant casts in an analyzing phase looks fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768526213
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768812256
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769610135
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134639/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565879134
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
Review comment:
yea
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565867620
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
+ } else {
+ // For view created before Spark 2.2.0, the view text is already fully qualified, the plan
+ // output is the same with the view output.
+ parsedPlan
Review comment:
For the issue "the schema of the referenced table/view is changed ...", doesn't this also suffer from it too? The view text is fully qualified doesn't mean it has no problem that the referenced table/view changes schema. Isn't?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -845,17 +845,33 @@ class SessionCatalog(
}
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
Review comment:
Hm? Is the comment "For view queries like `SELECT * FROM t`..." copied in this PR? I don't see its original place here.
This code seems copied from `EliminateView`, but its original comment is different. The `EliminateView`'s comment is more about resolution of attribute of view text.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -844,18 +844,41 @@ class SessionCatalog(
}
}
+ def getTempViewSchema(plan: LogicalPlan): StructType = {
+ plan match {
+ case viewInfo: TemporaryViewRelation => viewInfo.tableMeta.schema
+ case v => v.schema
+ }
+ }
+
private def fromCatalogTable(metadata: CatalogTable, isTempView: Boolean): View = {
- val viewText = metadata.viewText.getOrElse(sys.error("Invalid view without text."))
+ val viewText = metadata.viewText.getOrElse {
+ throw new IllegalStateException("Invalid view without text.")
+ }
val viewConfigs = metadata.viewSQLConfigs
- val viewPlan =
+ val parsedPlan =
SQLConf.withExistingConf(View.effectiveSQLConf(viewConfigs, isTempView = isTempView)) {
parser.parsePlan(viewText)
}
- View(
- desc = metadata,
- isTempView = isTempView,
- output = metadata.schema.toAttributes,
- child = viewPlan)
+ val viewColumnNames = metadata.viewQueryColumnNames
+ val viewPlan = if (viewColumnNames.nonEmpty) {
+ assert(viewColumnNames.length == metadata.schema.length)
+ // For view queries like `SELECT * FROM t`, the schema of the referenced table/view may
+ // change after the view has been created. We need to add an extra SELECT to pick the columns
+ // according to the recorded column names (to get the correct view column ordering and omit
+ // the extra columns that we don't require), add UpCast (to make sure the type change is
+ // safe) and Alias according to the schema in the catalog.
+ val projectList = viewColumnNames.zip(metadata.schema).map { case (col, field) =>
+ Alias(UpCast(UnresolvedAttribute.quoted(col), field.dataType), field.name)(
+ explicitMetadata = Some(field.metadata))
+ }
+ Project(projectList, parsedPlan)
Review comment:
If the child plan's output is same as view's schema, this projection will be removed by optimization, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769073585
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39204/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769176972
**[Test build #134621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134621/testReport)** for PR 31368 at commit [`2587e53`](https://github.com/apache/spark/commit/2587e538e6f76e214a9d26685526d43f97882395).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31368:
URL: https://github.com/apache/spark/pull/31368#discussion_r565562668
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
##########
@@ -655,28 +654,6 @@ class AnalysisSuite extends AnalysisTest with Matchers {
}
}
- test("SPARK-25691: AliasViewChild with different nullabilities") {
Review comment:
This test is not needed anymore because `EliminateView` is super simple now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768924274
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768909581
**[Test build #134605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134605/testReport)** for PR 31368 at commit [`dfc9d9d`](https://github.com/apache/spark/commit/dfc9d9d0921c81fde222107d3b4b35c5bb180c37).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768897083
**[Test build #134588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134588/testReport)** for PR 31368 at commit [`2c66b39`](https://github.com/apache/spark/commit/2c66b395166c56dc3ca1b867bf2b1af06e6a6d05).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769217884
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-768812243
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31368:
URL: https://github.com/apache/spark/pull/31368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769613707
GA passed, merging to master, thanks for the review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31368: [SPARK-34269][SQL] Simplify SQL view resolution
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31368:
URL: https://github.com/apache/spark/pull/31368#issuecomment-769055424
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134616/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org