You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/20 21:03:21 UTC
[GitHub] [spark] anchovYu opened a new pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
anchovYu opened a new pull request #29497:
URL: https://github.com/apache/spark/pull/29497
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690013117
**[Test build #128490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128490/testReport)** for PR 29497 at commit [`5877edf`](https://github.com/apache/spark/commit/5877edf048312734e5320965bf76657c2b04e27a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-678010288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727775302
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-678424616
> > standardization of error messages and its maintenance.
>
> How do we plan this?
The whole community needs to work on this together, define a guide for error messages and gradually improve the error messages in the future releases.
Grouping the messages in a single file should not be blocked by the guide and message improvement. Basically, these PRs are like code cleaning and refactoring. This effort will help us audit the error messages before each release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-679242942
I'm +1 to this idea, error message is super important to end-users, as it tells them what went wrong and how to fix it. It's easier to audit them if they are grouped together.
We need a clear way to organize it. This PR proposes `org.apache.spark.sql.CatalystErrors` for sql/catalyst, but how about sql/core, sql/hive and thriftserver? In general, there are 2 kinds of errors: query compilation error and query execution error, shall we group them separately?
one idea is to use different package names for different modules:
```
org.apache.spark.sql.catalyst.QueryCompilationErrors
org.apache.spark.sql.catalyst.QueryExecutionErrors
org.apache.spark.sql.QueryCompilationErrors
org.apache.spark.sql.QueryExecutionErrors
org.apache.spark.sql.hive.QueryCompilationErrors
org.apache.spark.sql.hive.QueryExecutionErrors
...
```
I'm open to other ideas as well. cc @maropu @viirya @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727870316
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-679549416
This idea sounds okay. I just have few questions.
1. Is it only for certain exceptions or all exceptions? For example, this PR targets only `AnalysisException`.
2. How to organize them? query compilation/query execution + package seems okay, but is there possibly unclear cases? For example, `AnalysisException` is also thrown in `RunnableCommand`, is it query compilation error and query execution error?
3. Is a top level object of error message too big to hold all exceptions in the module? For example `org.apache.spark.sql.QueryCompilationErrors` and `org.apache.spark.sql.QueryExecutionErrors` might contain many exceptions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727850717
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35743/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-731881187
late LGTM. Thank you all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677940097
cc @cloud-fan @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727806593
**[Test build #131140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131140/testReport)** for PR 29497 at commit [`645d81b`](https://github.com/apache/spark/commit/645d81bb4622c32119adab7c21c18ea3cce14fdb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677904563
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727809699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-728000650
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727996985
**[Test build #131144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131144/testReport)** for PR 29497 at commit [`645d81b`](https://github.com/apache/spark/commit/645d81bb4622c32119adab7c21c18ea3cce14fdb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474602580
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
Review comment:
It looks nice, @gatorsmile. I think we might be able to add a new rule in `scalastyle` for forbidding the use of `AnalysisException` in the the other files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-678009770
**[Test build #127708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127708/testReport)** for PR 29497 at commit [`29451f9`](https://github.com/apache/spark/commit/29451f92426afb8aaf2fc15c5dda144f93252b51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474541485
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object CatalystErrors {
+ def groupingIDMismatchError(groupingID: GroupingID, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Columns of grouping_id (${groupingID.groupByExprs.mkString(",")}) " +
+ s"does not match grouping columns (${groupByExprs.mkString(",")})")
+ }
+
+ def groupingColInvalidError(groupingCol: Expression, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Column of grouping ($groupingCol) can't be found " +
+ s"in grouping columns ${groupByExprs.mkString(",")}")
+ }
+
+ def groupingSizeTooLargeError(sizeLimit: Int): Throwable = {
+ new AnalysisException(
+ s"Grouping sets size cannot be greater than $sizeLimit")
+ }
+
+ def unorderablePivotColError(pivotCol: Expression): Throwable = {
+ new AnalysisException(
+ s"Invalid pivot column '$pivotCol'. Pivot columns must be comparable."
+ )
+ }
+
+ def nonliteralPivotValError(pivotVal: Expression): Throwable = {
+ new AnalysisException(
+ s"Literal expressions required for pivot values, found '$pivotVal'")
+ }
+
+ def pivotValDataTypeMismatchError(pivotVal: Expression, pivotCol: Expression): Throwable = {
+ new AnalysisException(
+ s"Invalid pivot value '$pivotVal': " +
+ s"value data type ${pivotVal.dataType.simpleString} does not match " +
+ s"pivot column data type ${pivotCol.dataType.catalogString}")
+ }
+
+ def unsupportedIfNotExistsError(tableName: String): Throwable = {
+ new AnalysisException(
+ s"Cannot write, IF NOT EXISTS is not supported for table: $tableName")
+ }
+
+ def nonPartitionColError(partitionName: String): Throwable = {
+ new AnalysisException(
+ s"PARTITION clause cannot contain a non-partition column name: $partitionName")
+ }
+
+ def addStaticValToUnknownColError(staticName: String): Throwable = {
+ new AnalysisException(
+ s"Cannot add static value for unknown column: $staticName")
+ }
+
+ def unknownStaticPartitionColError(name: String): Throwable = {
+ new AnalysisException(s"Unknown static partition column: $name")
+ }
+
+ def nestedGeneratorError(trimmedNestedGenerator: Expression): Throwable = {
+ new AnalysisException(
+ "Generators are not supported when it's nested in " +
+ "expressions, but got: " + toPrettySQL(trimmedNestedGenerator))
+ }
+
+ def moreThanOneGeneratorError(generators: Seq[Expression], clause: String): Throwable = {
+ new AnalysisException(
+ s"Only one generator allowed per $clause clause but found " +
+ generators.size + ": " + generators.map(toPrettySQL).mkString(", "))
+ }
+
+ def generatorOutsideSelectError(plan: LogicalPlan): Throwable = {
+ new AnalysisException(
+ "Generators are not supported outside the SELECT clause, but " +
+ "got: " + plan.simpleString(SQLConf.get.maxToStringFields))
+ }
+
+ def legacyStoreAssignmentPolicyError(): Throwable = {
+ val configKey = SQLConf.STORE_ASSIGNMENT_POLICY.key
+ new AnalysisException(
+ s"LEGACY store assignment policy is disallowed in Spark data source V2. " +
Review comment:
nit: let's remove leading `s` where it doesn't need.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727809951
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690952463
> that they are different package names + same object name.
This is a good point. Is it possible that we put all the error messages in the catalyst module? Other modules depend on catalyst and can access error messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677935441
**[Test build #127708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127708/testReport)** for PR 29497 at commit [`29451f9`](https://github.com/apache/spark/commit/29451f92426afb8aaf2fc15c5dda144f93252b51).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727806593
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727762739
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35736/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-728000650
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-714050304
> In the ideal case, all exception messages should be grouped for easy maintenance and auditing. This PR first starts from the AnalyzerExceptions in Catalyst.
And yeah, if we want to divide these exceptions in groups, then the mapping rule, and how to divide so that contributors can easily follow the rule to commit is a problem. And, another concern of the QueryCompilationErrors and QueryExecutionErrors is that they are different package names + same object name. When developers would like to call functions of compilation errors from different components, they may have to write the full package name.
>> This is a good point. Is it possible that we put all the error messages in the catalyst module? Other modules depend on catalyst and can access error messages.
Any update? I think its okay to target at grouping `AnalyzerExceptions` first in this PR and follow the @cloud-fan suggestion above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727775302
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677904563
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690013634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690035379
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-731460463
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690952463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727850742
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677935863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690952463
> that they are different package names + same object name.
This is a good point. Is it possible that we put all the error messages in the catalyst module? Other modules depend on catalyst and can access error messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727818488
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-679868589
> one idea is to use different package names for different modules:
Ah, I see. Splitting exceptions into multiple fine-graind groups looks a nice idea.
> How to organize them? query compilation/query execution + package seems okay, but is there possibly unclear cases? For example, AnalysisException is also thrown in RunnableCommand, is it query compilation error and query execution error?
I agree to the @viirya suggestion and I think we need a simple but clear rule to categorize exceptions into groups so that developers do not get confused. Probably, we can follow a simple mapping based on exception classes? e.g.,
```
AnalyssiException => QueryCompilationErrors
SparkException, RuntimeException(UnsupportedOperationException, IllegalStateException, ...) => QueryExecutionErrors
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r523946122
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
Review comment:
`Object for grouping all error messages of the query compilation.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741355
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727850763
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35743/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727870289
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35747/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690013634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
anchovYu commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474850404
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object CatalystErrors {
+ def groupingIDMismatchError(groupingID: GroupingID, groupByExprs: Seq[Expression]): Throwable = {
Review comment:
This is a summary sheet of raw exceptions thrown in Catalyst of branch-3.0: [OSS Catalyst Exception Messages: Branch-3.0](https://docs.google.com/spreadsheets/d/10HmOs-RIRaur2b_zLuOpRXC2b45KfpHwTBpQyalTV-s/edit?usp=sharing). Apart from `AnalysisException` and its sub-exceptions, there are a lot of other exceptions, for example, `IllegalArgumentException` and `UnsupportedOperationException`.
We generally have two ways to group these messages:
* By component. All messages from a single components, no matter the exception type, are grouped into one exception file.
* By exception type. All messages of a single exception type are created in the corresponding exception object. It is great because by calling `AnalysisException.groupingIDMismatchError` we know the exception type this function throws. But this approach has a problem: those Java/Scala internal exception type cannot follow this way.
Due to the small problem of the latter one, I choose the first one of grouping all exception messages from Catalyst in one file. Or do you have any thoughts on that? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727740247
**[Test build #131131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131131/testReport)** for PR 29497 at commit [`032c916`](https://github.com/apache/spark/commit/032c9160b3cbaefa2be2a3ba7c4c57530a3e4d6c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727821409
**[Test build #131144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131144/testReport)** for PR 29497 at commit [`645d81b`](https://github.com/apache/spark/commit/645d81bb4622c32119adab7c21c18ea3cce14fdb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r500747049
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object CatalystErrors {
Review comment:
rename CatalystErrors => QueryCompilationErrors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-678010288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727745405
**[Test build #131133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131133/testReport)** for PR 29497 at commit [`f391721`](https://github.com/apache/spark/commit/f39172149a266072128f5adf8308cad55efe2d45).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727850742
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727834734
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35743/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727821409
**[Test build #131144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131144/testReport)** for PR 29497 at commit [`645d81b`](https://github.com/apache/spark/commit/645d81bb4622c32119adab7c21c18ea3cce14fdb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727870316
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29497:
URL: https://github.com/apache/spark/pull/29497
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690035389
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128490/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690035379
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r528431634
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages of the query compilation.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
Review comment:
`AnalysisExcpetions` -> `AnalysisExceptions`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474449082
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
Review comment:
We need to do all of them, I think. This is just an example PR.
This can help us do error message auditing. In the future, we can review this file, improve the error message quality, and make them unified and standard in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741674
**[Test build #131131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131131/testReport)** for PR 29497 at commit [`032c916`](https://github.com/apache/spark/commit/032c9160b3cbaefa2be2a3ba7c4c57530a3e4d6c).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727809699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anchovYu commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
anchovYu commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690019832
> 1. Is it only for certain exceptions or all exceptions? For example, this PR targets only AnalysisException.
In the ideal case, all exception messages should be grouped for easy maintenance and auditing. This PR first starts from the `AnalyzerException`s in Catalyst.
And yeah, if we want to divide these exceptions in groups, then the mapping rule, and how to divide so that contributors can easily follow the rule to commit is a problem. And, another concern of the `QueryCompilationErrors` and `QueryExecutionErrors` is that they are different package names + same object name. When developers would like to call functions of compilation errors from different components, they may have to write the full package name.
Grouping exceptions in a single file will necessarily explode the file size; but I think the process is that first we group them, then we look for ways to optimize them, for example, find the duplicate error messages and combine them in one function, or, divide them into smaller and refined groups.
Open to all ideas and great thanks for your review! cc @cloud-fan @maropu @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727740247
**[Test build #131131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131131/testReport)** for PR 29497 at commit [`032c916`](https://github.com/apache/spark/commit/032c9160b3cbaefa2be2a3ba7c4c57530a3e4d6c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727809960
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131133/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727775287
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35736/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677904063
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677904063
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741355
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-731442169
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727741699
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131131/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r523946524
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object QueryCompilationErrors {
+ def groupingIDMismatchError(groupingID: GroupingID, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Columns of grouping_id (${groupingID.groupByExprs.mkString(",")}) " +
+ s"does not match grouping columns (${groupByExprs.mkString(",")})")
+ }
+
+ def groupingColInvalidError(groupingCol: Expression, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Column of grouping ($groupingCol) can't be found " +
+ s"in grouping columns ${groupByExprs.mkString(",")}")
+ }
+
+ def groupingSizeTooLargeError(sizeLimit: Int): Throwable = {
+ new AnalysisException(
+ s"Grouping sets size cannot be greater than $sizeLimit")
+ }
+
+ def unorderablePivotColError(pivotCol: Expression): Throwable = {
+ new AnalysisException(
+ s"Invalid pivot column '$pivotCol'. Pivot columns must be comparable."
+ )
+ }
+
+ def nonliteralPivotValError(pivotVal: Expression): Throwable = {
Review comment:
`nonLiteral...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690035168
**[Test build #128490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128490/testReport)** for PR 29497 at commit [`5877edf`](https://github.com/apache/spark/commit/5877edf048312734e5320965bf76657c2b04e27a).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727809569
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677934777
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-690013117
**[Test build #128490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128490/testReport)** for PR 29497 at commit [`5877edf`](https://github.com/apache/spark/commit/5877edf048312734e5320965bf76657c2b04e27a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29497: [SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r528432337
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages of the query compilation.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object QueryCompilationErrors {
+ def groupingIDMismatchError(groupingID: GroupingID, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Columns of grouping_id (${groupingID.groupByExprs.mkString(",")}) " +
+ s"does not match grouping columns (${groupByExprs.mkString(",")})")
+ }
+
+ def groupingColInvalidError(groupingCol: Expression, groupByExprs: Seq[Expression]): Throwable = {
+ new AnalysisException(
+ s"Column of grouping ($groupingCol) can't be found " +
+ s"in grouping columns ${groupByExprs.mkString(",")}")
+ }
+
+ def groupingSizeTooLargeError(sizeLimit: Int): Throwable = {
+ new AnalysisException(
+ s"Grouping sets size cannot be greater than $sizeLimit")
Review comment:
Although this is related to the original message, maybe, `set's size` or `sets' size` is better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677935863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-727852949
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35747/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474321198
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
+ */
+object CatalystErrors {
+ def groupingIDMismatchError(groupingID: GroupingID, groupByExprs: Seq[Expression]): Throwable = {
Review comment:
nit: how about moving these methods into `object AnalysisException`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r500746905
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
Review comment:
move org.apache.spark.sql => org.apache.spark.sql.errors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-677935441
**[Test build #127708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127708/testReport)** for PR 29497 at commit [`29451f9`](https://github.com/apache/spark/commit/29451f92426afb8aaf2fc15c5dda144f93252b51).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29497:
URL: https://github.com/apache/spark/pull/29497#issuecomment-678125244
> standardization of error messages and its maintenance.
How do we plan this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29497: [WIP][SPARK-32670][SQL]Group exception messages in Catalyst Analyzer in one file
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29497:
URL: https://github.com/apache/spark/pull/29497#discussion_r474322596
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/CatalystErrors.scala
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.expressions.{Expression, GroupingID}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.toPrettySQL
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StructType}
+
+/**
+ * Object for grouping all error messages in catalyst.
+ * Currently it includes all AnalysisExcpetions created and thrown directly in
+ * org.apache.spark.sql.catalyst.analysis.Analyzer.
Review comment:
Just a question; did you propose to group **all** the analysis exception here? I think we already have too many places (~500) where the analysis exceptions used though... https://gist.github.com/maropu/29bbfa9a93c41bd4ba1e0eaa038af087
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org