You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/15 10:44:56 UTC
[GitHub] [spark] MaxGekk opened a new pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
MaxGekk opened a new pull request #34600:
URL: https://github.com/apache/spark/pull/34600
### What changes were proposed in this pull request?
In the PR, I propose to allow ANSI intervals: year-month and day-time intervals in the `ALTER TABLE .. ADD COLUMNS` command for tables in v1 and v2 In-Memory catalogs (v1 Hive external catalog doesn't support the command). Also added an unified test suite to migrate related tests in the future.
### Why are the changes needed?
To improve user experience with Spark SQL. After the changes, users will be able to add columns with ANSI intervals instead of dropping and creating new table.
### Does this PR introduce _any_ user-facing change?
In some sense, yes. After the changes, the command doesn't output any error message.
### How was this patch tested?
By running new test suite:
```
$ build/sbt "test:testOnly *AlterTableAddColumnsSuite"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968951313
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969022015
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968844699
@cloud-fan FYI, Here is the ticket for fully unified tests of `ALTER TABLE .. ADD COLUMNS` https://issues.apache.org/jira/browse/SPARK-37045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969335592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145241/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968813301
**[Test build #145230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145230/testReport)** for PR 34600 at commit [`81fef07`](https://github.com/apache/spark/commit/81fef07ce81268db612d1502f8e7f1b534a619bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969083635
**[Test build #145241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145241/testReport)** for PR 34600 at commit [`df18813`](https://github.com/apache/spark/commit/df1881369524217e7d73eae95e85cef386c4e15d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969802628
I cloned `MaxGekk:add-columns-ansi-interval` and run GA jobs on my repository, and `docker integration tests` passed.
https://github.com/sarutak/spark/runs/4219609621?check_suite_focus=true
So, I think the `docker integration tests` failure on your repository is not related to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968965848
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145230/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969074516
> We do not have one suite for hive catalog?
@cloud-fan I have added one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969166621
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969192665
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34600:
URL: https://github.com/apache/spark/pull/34600#discussion_r749251936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableAddColumnsSuiteBase.scala
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.time.{Duration, Period}
+
+import org.apache.spark.sql.{QueryTest, Row}
+
+/**
+ * This base suite contains unified tests for the `ALTER TABLE .. ADD COLUMNS` command that
+ * check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v2.AlterTableAddColumnsSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.AlterTableAddColumnsSuiteBase`
+ * - V1 In-Memory catalog:
+ * `org.apache.spark.sql.execution.command.v1.AlterTableAddColumnsSuite`
+ * - V1 Hive External catalog:
Review comment:
As you mention in the description, Hive External catalog doesn't support `ALTER TABLE .. ADD COLUMNS` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968842172
@cloud-fan @sarutak @Peng-Lei @AngersZhuuuu Could you review this PR, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969192665
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968890385
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49700/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969330673
**[Test build #145241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145241/testReport)** for PR 34600 at commit [`df18813`](https://github.com/apache/spark/commit/df1881369524217e7d73eae95e85cef386c4e15d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969083635
**[Test build #145241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145241/testReport)** for PR 34600 at commit [`df18813`](https://github.com/apache/spark/commit/df1881369524217e7d73eae95e85cef386c4e15d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968960239
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968964950
**[Test build #145230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145230/testReport)** for PR 34600 at commit [`81fef07`](https://github.com/apache/spark/commit/81fef07ce81268db612d1502f8e7f1b534a619bd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34600:
URL: https://github.com/apache/spark/pull/34600#discussion_r749251936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableAddColumnsSuiteBase.scala
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.time.{Duration, Period}
+
+import org.apache.spark.sql.{QueryTest, Row}
+
+/**
+ * This base suite contains unified tests for the `ALTER TABLE .. ADD COLUMNS` command that
+ * check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v2.AlterTableAddColumnsSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.AlterTableAddColumnsSuiteBase`
+ * - V1 In-Memory catalog:
+ * `org.apache.spark.sql.execution.command.v1.AlterTableAddColumnsSuite`
+ * - V1 Hive External catalog:
Review comment:
As you mention in the description, Hive External catalog doesn't support `ALTER TABLE .. ADD COLUMNS` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968898404
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968906738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49700/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968813301
**[Test build #145230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145230/testReport)** for PR 34600 at commit [`81fef07`](https://github.com/apache/spark/commit/81fef07ce81268db612d1502f8e7f1b534a619bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968906738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49700/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968848478
> v1 Hive external catalog doesn't support ANSI intervals
I think Hive external catalog supports ANSI intervals. In this case, the format of metadata is not Hive compatible but Spark specific.
```
spark-sql> CREATE TABLE tbl2(ym INTERVAL YEAR TO MONTH, dt INTERVAL DAY TO SECOND) using Parquet;
21/11/15 21:05:08 WARN HiveExternalCatalog: Hive incompatible types found: interval year to month, interval day to second. Persisting data source table `default`.`tbl2` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.
spark-sql> INSERT INTO tbl2 VALUES(INTERVAL '1-2' YEAR TO MONTH, INTERVAL '1 2:3:4' DAY TO SECOND);
spark-sql> SELECT * FROM tbl2;
1-2 1 02:03:04.000000000
Time taken: 0.381 seconds, Fetched 1 row(s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968965848
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145230/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969021196
**[Test build #145233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145233/testReport)** for PR 34600 at commit [`84e9a0e`](https://github.com/apache/spark/commit/84e9a0ede5785d4e4ea3f3ec634f49494a82b196).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968845388
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49700/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969335592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145241/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #34600:
URL: https://github.com/apache/spark/pull/34600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968839386
> v1 Hive external catalog doesn't support the command
v1 Hive external catalog supports `ALTER TABLE ADD COLUMN` doesn't it?
```
spark-sql> CREATE TABLE tbl1(x INT) using Parquet;
spark-sql> ALTER TABLE tbl1 ADD COLUMN (y STRING);
spark-sql> DESCRIBE tbl1;
x int
y string
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968862360
**[Test build #145233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145233/testReport)** for PR 34600 at commit [`84e9a0e`](https://github.com/apache/spark/commit/84e9a0ede5785d4e4ea3f3ec634f49494a82b196).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968960239
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49703/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968840706
> v1 Hive external catalog supports ALTER TABLE ADD COLUMN doesn't it?
Thanks. I have corrected the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969022015
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145233/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34600:
URL: https://github.com/apache/spark/pull/34600#discussion_r749416304
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableAddColumnsSuiteBase.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.time.{Duration, Period}
+
+import org.apache.spark.sql.{QueryTest, Row}
+
+/**
+ * This base suite contains unified tests for the `ALTER TABLE .. ADD COLUMNS` command that
+ * check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v2.AlterTableAddColumnsSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.AlterTableAddColumnsSuiteBase`
+ * - V1 In-Memory catalog:
Review comment:
We do not have one suite for hive catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in v2 `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-968862360
**[Test build #145233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145233/testReport)** for PR 34600 at commit [`84e9a0e`](https://github.com/apache/spark/commit/84e9a0ede5785d4e4ea3f3ec634f49494a82b196).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969123384
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969958940
Merging to master. Thank you, @sarutak and @cloud-fan for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34600:
URL: https://github.com/apache/spark/pull/34600#issuecomment-969802628
I cloned `MaxGekk:add-columns-ansi-interval` and run GA jobs on my repository, and `docker integration tests` passed.
https://github.com/sarutak/spark/runs/4219609621?check_suite_focus=true
So, I think the `docker integration tests` failure on your repository is not related to this change.
https://github.com/MaxGekk/spark/runs/4214187254?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org