You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jiangxb1987 <gi...@git.apache.org> on 2018/02/28 15:38:13 UTC

[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

GitHub user jiangxb1987 opened a pull request:

    https://github.com/apache/spark/pull/20696

    [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table

    ## What changes were proposed in this pull request?
    
    The following query doesn't work as expected:
    ```
    CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING)
    LOCATION 'sql/core/spark-warehouse/ext_table';
    ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment";
    DESC ext_table;
    ```
    The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`.
    
    ## How was this patch tested?
    
    Updated `DDLSuite.testChangeColumn`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jiangxb1987/spark alterColumnComment

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20696.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20696
    
----
commit 92b2f317aca1c8fe96e4448c8d6dcedce02c9429
Author: Xingbo Jiang <xi...@...>
Date:   2018-02-28T15:29:19Z

    support change column comment for external hive table

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1339/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #87786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87786/testReport)** for PR 20696 at commit [`92b2f31`](https://github.com/apache/spark/commit/92b2f317aca1c8fe96e4448c8d6dcedce02c9429).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1194/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #87842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87842/testReport)** for PR 20696 at commit [`6035d7e`](https://github.com/apache/spark/commit/6035d7e102c7e8d004e0c967ca7e4904f6053472).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r172354978
  
    --- Diff: sql/core/src/test/resources/sql-tests/results/change-column.sql.out ---
    @@ -291,16 +291,25 @@ ALTER TABLE partition_table PARTITION (c = 1) CHANGE COLUMN a new_a INT
     
     
     -- !query 30
    -DROP TABLE test_change
    +ALTER TABLE partition_table CHANGE COLUMN c c INT COMMENT 'this is column C'
     -- !query 30 schema
     struct<>
     -- !query 30 output
    -
    +org.apache.spark.sql.AnalysisException
    +Invalid column reference 'c', table data schema is 'StructType(StructField(a,IntegerType,true), StructField(b,StringType,true))';
    --- End diff --
    
    how about 
    ```
    Can't find column `c` given table data columns [`a`, `b`]
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88031/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r171462718
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
    @@ -314,8 +314,8 @@ case class AlterTableChangeColumnCommand(
         val resolver = sparkSession.sessionState.conf.resolver
         DDLUtils.verifyAlterTableType(catalog, table, isView = false)
     
    -    // Find the origin column from schema by column name.
    -    val originColumn = findColumnByName(table.schema, columnName, resolver)
    +    // Find the origin column from dataSchema by column name.
    +    val originColumn = findColumnByName(table.dataSchema, columnName, resolver)
    --- End diff --
    
    Do we have a negative test case to cover that?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87842/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r171439915
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
    @@ -314,8 +314,8 @@ case class AlterTableChangeColumnCommand(
         val resolver = sparkSession.sessionState.conf.resolver
         DDLUtils.verifyAlterTableType(catalog, table, isView = false)
     
    -    // Find the origin column from schema by column name.
    -    val originColumn = findColumnByName(table.schema, columnName, resolver)
    +    // Find the origin column from dataSchema by column name.
    +    val originColumn = findColumnByName(table.dataSchema, columnName, resolver)
    --- End diff --
    
    As described in the comment, we only support altering non-partition columns:
    ```
     A command to change the column for a table, only support changing the comment of a non-partition
     column for now.
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20696


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r171317432
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
    @@ -314,8 +314,8 @@ case class AlterTableChangeColumnCommand(
         val resolver = sparkSession.sessionState.conf.resolver
         DDLUtils.verifyAlterTableType(catalog, table, isView = false)
     
    -    // Find the origin column from schema by column name.
    -    val originColumn = findColumnByName(table.schema, columnName, resolver)
    +    // Find the origin column from dataSchema by column name.
    +    val originColumn = findColumnByName(table.dataSchema, columnName, resolver)
    --- End diff --
    
    Only data schema? How about partitioning schema?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #87786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87786/testReport)** for PR 20696 at commit [`92b2f31`](https://github.com/apache/spark/commit/92b2f317aca1c8fe96e4448c8d6dcedce02c9429).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    thanks, merging to master/2.3!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #87842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87842/testReport)** for PR 20696 at commit [`6035d7e`](https://github.com/apache/spark/commit/6035d7e102c7e8d004e0c967ca7e4904f6053472).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r171623841
  
    --- Diff: sql/core/src/test/resources/sql-tests/results/change-column.sql.out ---
    @@ -291,16 +291,25 @@ ALTER TABLE partition_table PARTITION (c = 1) CHANGE COLUMN a new_a INT
     
     
     -- !query 30
    -DROP TABLE test_change
    +ALTER TABLE partition_table CHANGE COLUMN c c INT COMMENT 'this is column C'
     -- !query 30 schema
     struct<>
     -- !query 30 output
    -
    +org.apache.spark.sql.AnalysisException
    +Invalid column reference 'c', table data schema is 'StructType(StructField(a,IntegerType,true), StructField(b,StringType,true))';
    --- End diff --
    
    We need a better error message here.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1155/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #88031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88031/testReport)** for PR 20696 at commit [`48fc338`](https://github.com/apache/spark/commit/48fc338dc30720aa05e1871d69bad66ae2dfaa59).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20696#discussion_r171285332
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
    @@ -1597,6 +1597,7 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
         // Ensure that change column will preserve other metadata fields.
         sql("ALTER TABLE dbx.tab1 CHANGE COLUMN col1 col1 INT COMMENT 'this is col1'")
         assert(getMetadata("col1").getString("key") == "value")
    +    assert(getMetadata("col1").getString("comment") == "this is col1")
    --- End diff --
    
    I didn't verify the comment is correctly set to the column metadata, and that caused the issue. :(


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    cc @skliarpawlo @gatorsmile @cloud-fan 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    **[Test build #88031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88031/testReport)** for PR 20696 at commit [`48fc338`](https://github.com/apache/spark/commit/48fc338dc30720aa05e1871d69bad66ae2dfaa59).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20696
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87786/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org