You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/02 07:10:09 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
AngersZhuuuu opened a new pull request #31705:
URL: https://github.com/apache/spark/pull/31705
### What changes were proposed in this pull request?
In the PR, I propose to generate "stable" output attributes per the logical node of the DESCRIBE NAMESPACE command.
### Why are the changes needed?
This fixes the issue demonstrated by the example:
```
sql(s"CREATE NAMESPACE ns")
val description = sql(s"DESCRIBE NAMESPACE ns")
description.drop("name")
```
```
[info] org.apache.spark.sql.AnalysisException: Resolved attribute(s) name#74 missing from name#25,value#26 in operator !Project [name#74]. Attribute(s) with the same name appear in the operation: name. Please check if the right attribute(s) are used.;
[info] !Project [name#74]
[info] +- LocalRelation [name#25, value#26]
```
### Does this PR introduce _any_ user-facing change?
After this change user `drop()/add()` works well.
### How was this patch tested?
Added UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788969141
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40240/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r585395262
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -296,10 +296,14 @@ case class DropNamespace(
*/
case class DescribeNamespace(
namespace: LogicalPlan,
- extended: Boolean) extends Command {
+ extended: Boolean,
+ override val output: Seq[Attribute] = DescribeNamespace.getOutputAttr) extends Command {
override def children: Seq[LogicalPlan] = Seq(namespace)
- override def output: Seq[Attribute] = Seq(
+}
+
+object DescribeNamespace {
+ def getOutputAttr: Seq[Attribute] = Seq(
Review comment:
the v2 schema is weird: `name: string, value: string`. Shall we follow DESC COLUMN and use `info_name: string, info_value: string`?
v2 command is not stable yet and we can still change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788777284
**[Test build #135651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135651/testReport)** for PR 31705 at commit [`41584f5`](https://github.com/apache/spark/commit/41584f5eae33d5e0d10c72fa2bf6810aa2feca55).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789410119
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40266/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789935184
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135709/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789573783
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40281/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788739051
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40219/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789933910
**[Test build #135709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135709/testReport)** for PR 31705 at commit [`72345b2`](https://github.com/apache/spark/commit/72345b2351cd447c61b93cc186952e6ae526fdfd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789935184
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135709/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r585548271
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeCommandSchema.scala
##########
@@ -21,6 +21,12 @@ import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.types.{MetadataBuilder, StringType}
private[sql] object DescribeCommandSchema {
+ def describeNamespaceAttributes(): Seq[AttributeReference] = Seq(
+ AttributeReference("info_name", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "name of the column info").build())(),
Review comment:
Done
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeCommandSchema.scala
##########
@@ -21,6 +21,12 @@ import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.types.{MetadataBuilder, StringType}
private[sql] object DescribeCommandSchema {
+ def describeNamespaceAttributes(): Seq[AttributeReference] = Seq(
+ AttributeReference("info_name", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "name of the column info").build())(),
+ AttributeReference("info_value", StringType, nullable = true,
+ new MetadataBuilder().putString("comment", "value of the column info").build())())
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788712309
**[Test build #135639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135639/testReport)** for PR 31705 at commit [`358d1ec`](https://github.com/apache/spark/commit/358d1ece64a28fe8d4609ad7f8a871bec6cf06af).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789530721
**[Test build #135699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135699/testReport)** for PR 31705 at commit [`131c46e`](https://github.com/apache/spark/commit/131c46e55b44fac6433d5a9f6aa2f7f00fc6fd02).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788674007
FYI @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788849642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40231/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788739028
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40219/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789769112
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40291/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789530721
**[Test build #135699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135699/testReport)** for PR 31705 at commit [`131c46e`](https://github.com/apache/spark/commit/131c46e55b44fac6433d5a9f6aa2f7f00fc6fd02).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788712309
**[Test build #135639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135639/testReport)** for PR 31705 at commit [`358d1ec`](https://github.com/apache/spark/commit/358d1ece64a28fe8d4609ad7f8a871bec6cf06af).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789708861
**[Test build #135709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135709/testReport)** for PR 31705 at commit [`72345b2`](https://github.com/apache/spark/commit/72345b2351cd447c61b93cc186952e6ae526fdfd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788866575
**[Test build #135651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135651/testReport)** for PR 31705 at commit [`41584f5`](https://github.com/apache/spark/commit/41584f5eae33d5e0d10c72fa2bf6810aa2feca55).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789410135
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40266/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31705:
URL: https://github.com/apache/spark/pull/31705
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r586176440
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -365,18 +365,20 @@ class HiveCatalogedDDLSuite extends DDLSuite with TestHiveSingleton with BeforeA
test("Database Ownership") {
val catalog = spark.sessionState.catalog
- try {
- val db = "spark_29425_1"
- sql(s"CREATE DATABASE $db")
- assert(sql(s"DESCRIBE DATABASE EXTENDED $db")
- .where("database_description_item='Owner'")
Review comment:
can we use the new column name instead of turning on the legacy config?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789371876
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r612578056
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##########
@@ -179,8 +179,15 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
case UnsetViewProperties(ResolvedView(ident, _), keys, ifExists) =>
AlterTableUnsetPropertiesCommand(ident.asTableIdentifier, keys, ifExists, isView = true)
- case d @ DescribeNamespace(DatabaseInSessionCatalog(db), _) =>
- DescribeDatabaseCommand(db, d.extended)
+ case DescribeNamespace(DatabaseInSessionCatalog(db), extended, output) =>
+ val newOutput = if (conf.getConf(SQLConf.LEGACY_KEEP_COMMAND_OUTPUT_SCHEMA)) {
Review comment:
@AngersZhuuuu can we add one item in the migration guide for this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788739051
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40219/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r586179845
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -365,18 +365,20 @@ class HiveCatalogedDDLSuite extends DDLSuite with TestHiveSingleton with BeforeA
test("Database Ownership") {
val catalog = spark.sessionState.catalog
- try {
- val db = "spark_29425_1"
- sql(s"CREATE DATABASE $db")
- assert(sql(s"DESCRIBE DATABASE EXTENDED $db")
- .where("database_description_item='Owner'")
Review comment:
> can we use the new column name instead of turning on the legacy config?
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788927648
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135659/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788777284
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r585548984
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -296,14 +296,14 @@ case class DropNamespace(
*/
case class DescribeNamespace(
namespace: LogicalPlan,
- extended: Boolean) extends Command {
+ extended: Boolean,
+ override val output: Seq[Attribute] = DescribeNamespace.getOutputAttr) extends Command {
override def children: Seq[LogicalPlan] = Seq(namespace)
- override def output: Seq[Attribute] = Seq(
- AttributeReference("name", StringType, nullable = false,
- new MetadataBuilder().putString("comment", "name of the column").build())(),
- AttributeReference("value", StringType, nullable = true,
- new MetadataBuilder().putString("comment", "value of the column").build())())
+}
+
+object DescribeNamespace {
+ def getOutputAttr: Seq[Attribute] = DescribeCommandSchema.describeNamespaceAttributes()
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789740113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40291/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789769112
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40291/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789585752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40281/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789390698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135684/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789005659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40240/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789768380
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40291/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r585540653
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeCommandSchema.scala
##########
@@ -21,6 +21,12 @@ import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.types.{MetadataBuilder, StringType}
private[sql] object DescribeCommandSchema {
+ def describeNamespaceAttributes(): Seq[AttributeReference] = Seq(
+ AttributeReference("info_name", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "name of the column info").build())(),
Review comment:
name of the namespace info
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeCommandSchema.scala
##########
@@ -21,6 +21,12 @@ import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.types.{MetadataBuilder, StringType}
private[sql] object DescribeCommandSchema {
+ def describeNamespaceAttributes(): Seq[AttributeReference] = Seq(
+ AttributeReference("info_name", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "name of the column info").build())(),
+ AttributeReference("info_value", StringType, nullable = true,
+ new MetadataBuilder().putString("comment", "value of the column info").build())())
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789708861
**[Test build #135709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135709/testReport)** for PR 31705 at commit [`72345b2`](https://github.com/apache/spark/commit/72345b2351cd447c61b93cc186952e6ae526fdfd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789408551
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40266/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788927648
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135659/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788875841
**[Test build #135639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135639/testReport)** for PR 31705 at commit [`358d1ec`](https://github.com/apache/spark/commit/358d1ece64a28fe8d4609ad7f8a871bec6cf06af).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788849642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40231/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788996460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40240/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789005659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40240/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789493913
> it's a real failure in `DataSourceV2SQLSuite.DescribeNamespace using v2 catalog`
Updated, current should be ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789622984
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135699/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788883605
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r612893140
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##########
@@ -179,8 +179,15 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
case UnsetViewProperties(ResolvedView(ident, _), keys, ifExists) =>
AlterTableUnsetPropertiesCommand(ident.asTableIdentifier, keys, ifExists, isView = true)
- case d @ DescribeNamespace(DatabaseInSessionCatalog(db), _) =>
- DescribeDatabaseCommand(db, d.extended)
+ case DescribeNamespace(DatabaseInSessionCatalog(db), extended, output) =>
+ val newOutput = if (conf.getConf(SQLConf.LEGACY_KEEP_COMMAND_OUTPUT_SCHEMA)) {
Review comment:
Done https://github.com/apache/spark/pull/32155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r585541202
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -296,14 +296,14 @@ case class DropNamespace(
*/
case class DescribeNamespace(
namespace: LogicalPlan,
- extended: Boolean) extends Command {
+ extended: Boolean,
+ override val output: Seq[Attribute] = DescribeNamespace.getOutputAttr) extends Command {
override def children: Seq[LogicalPlan] = Seq(namespace)
- override def output: Seq[Attribute] = Seq(
- AttributeReference("name", StringType, nullable = false,
- new MetadataBuilder().putString("comment", "name of the column").build())(),
- AttributeReference("value", StringType, nullable = true,
- new MetadataBuilder().putString("comment", "value of the column").build())())
+}
+
+object DescribeNamespace {
+ def getOutputAttr: Seq[Attribute] = DescribeCommandSchema.describeNamespaceAttributes()
Review comment:
shall we define the output here instead of `DescribeCommandSchema`? It's only used once here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788732946
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40219/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789421636
it's a real failure in `DataSourceV2SQLSuite.DescribeNamespace using v2 catalog`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789390698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135684/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789622984
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135699/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31705:
URL: https://github.com/apache/spark/pull/31705#discussion_r586178173
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -365,18 +365,20 @@ class HiveCatalogedDDLSuite extends DDLSuite with TestHiveSingleton with BeforeA
test("Database Ownership") {
val catalog = spark.sessionState.catalog
- try {
- val db = "spark_29425_1"
- sql(s"CREATE DATABASE $db")
- assert(sql(s"DESCRIBE DATABASE EXTENDED $db")
- .where("database_description_item='Owner'")
Review comment:
> can we use the new column name instead of turning on the legacy config?
TBH, I am confused why we use such a strange col name before, can we just ignore this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-788883605
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789622325
**[Test build #135699 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135699/testReport)** for PR 31705 at commit [`131c46e`](https://github.com/apache/spark/commit/131c46e55b44fac6433d5a9f6aa2f7f00fc6fd02).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789551055
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40281/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-790300849
thanks, merging to master/3.1!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789585752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40281/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31705: [SPARK-34577][SQL] Fix drop/add columns to a dataset of `DESCRIBE NAMESPACE`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31705:
URL: https://github.com/apache/spark/pull/31705#issuecomment-789410135
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40266/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org