You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/08/22 14:04:31 UTC

[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

GitHub user cloud-fan opened a pull request:

    https://github.com/apache/spark/pull/14752

    [SPARK-17186][SQL] remove catalog table type INDEX

    ## What changes were proposed in this pull request?
    
    Actually Spark SQL doesn't support index, the catalog table type `INDEX` is from Hive. However, most operations in Spark SQL can't handle index table, e.g. create table, alter table, etc.
    
    Logically index table should be invisible to end users, and Hive also generates special table name for index table to avoid users accessing it directly. Hive has special SQL syntax to create/show/drop index tables.
    
    At Spark SQL side, although we can describe index table directly, but the result is unreadable, we should use the dedicated SQL syntax to do it(e.g. `SHOW INDEX ON tbl`). Spark SQL can also read index table directly, but the result is always empty.(Can hive read index table directly?)
    
    This PR remove the table type `INDEX`, to make it clear that Spark SQL doesn't support index currently.
    
    ## How was this patch tested?
    
    existing tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cloud-fan/spark minor2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/14752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14752
    
----
commit d2bc794767eef9a357c7cb77cdef017403516cb9
Author: Wenchen Fan <we...@databricks.com>
Date:   2016-08-22T13:41:18Z

    remove catalog table type INDEX

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    cc @yhuai @gatorsmile 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64245/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64253/consoleFull)** for PR 14752 at commit [`d1e83d2`](https://github.com/apache/spark/commit/d1e83d2ace803b51ea3d25c575b84e1af6bde353).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64245/consoleFull)** for PR 14752 at commit [`688802b`](https://github.com/apache/spark/commit/688802b7ec40498aba309ebb4ce0a1c667456635).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14752#discussion_r75767908
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ---
    @@ -379,8 +379,9 @@ private[hive] class HiveClientImpl(
             tableType = h.getTableType match {
               case HiveTableType.EXTERNAL_TABLE => CatalogTableType.EXTERNAL
               case HiveTableType.MANAGED_TABLE => CatalogTableType.MANAGED
    -          case HiveTableType.INDEX_TABLE => CatalogTableType.INDEX
               case HiveTableType.VIRTUAL_VIEW => CatalogTableType.VIEW
    +          case HiveTableType.INDEX_TABLE =>
    +            throw new AnalysisException("Hive index table is not supported.")
    --- End diff --
    
    can this be an error triggered error? if not i'd throw a different exception since at this point it's a bug in spark.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64253/consoleFull)** for PR 14752 at commit [`d1e83d2`](https://github.com/apache/spark/commit/d1e83d2ace803b51ea3d25c575b84e1af6bde353).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    The changes look reasonable. We just need to completely remove them from the code base.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14752#discussion_r75782141
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ---
    @@ -379,8 +379,9 @@ private[hive] class HiveClientImpl(
             tableType = h.getTableType match {
               case HiveTableType.EXTERNAL_TABLE => CatalogTableType.EXTERNAL
               case HiveTableType.MANAGED_TABLE => CatalogTableType.MANAGED
    -          case HiveTableType.INDEX_TABLE => CatalogTableType.INDEX
               case HiveTableType.VIRTUAL_VIEW => CatalogTableType.VIEW
    +          case HiveTableType.INDEX_TABLE =>
    +            throw new AnalysisException("Hive index table is not supported.")
    --- End diff --
    
    This can be triggered. Users can get the special index table name by `SHOW TABLES` and read it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64205/consoleFull)** for PR 14752 at commit [`d2bc794`](https://github.com/apache/spark/commit/d2bc794767eef9a357c7cb77cdef017403516cb9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64253/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14752#discussion_r76065946
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ---
    @@ -379,8 +379,9 @@ private[hive] class HiveClientImpl(
             tableType = h.getTableType match {
               case HiveTableType.EXTERNAL_TABLE => CatalogTableType.EXTERNAL
               case HiveTableType.MANAGED_TABLE => CatalogTableType.MANAGED
    -          case HiveTableType.INDEX_TABLE => CatalogTableType.INDEX
               case HiveTableType.VIRTUAL_VIEW => CatalogTableType.VIEW
    +          case HiveTableType.INDEX_TABLE =>
    +            throw new AnalysisException("Hive index table is not supported.")
    --- End diff --
    
    Sure, I can do it today. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64205/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14752#discussion_r75911904
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ---
    @@ -379,8 +379,9 @@ private[hive] class HiveClientImpl(
             tableType = h.getTableType match {
               case HiveTableType.EXTERNAL_TABLE => CatalogTableType.EXTERNAL
               case HiveTableType.MANAGED_TABLE => CatalogTableType.MANAGED
    -          case HiveTableType.INDEX_TABLE => CatalogTableType.INDEX
               case HiveTableType.VIRTUAL_VIEW => CatalogTableType.VIEW
    +          case HiveTableType.INDEX_TABLE =>
    +            throw new AnalysisException("Hive index table is not supported.")
    --- End diff --
    
    Any function who invokes `getTableOption` might trigger it.
    
    For example, when we create a table that has the same name as an existing Index table, the message we will get is 
    ```
    Hive index table is not supported
    ```
    This might be confusing to external users, because they are not creating a Hive index table. 
    
    In Hive, the error message we will get is
    ```
    hive> create table default__t21_id1__ (col1 int);
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table default__t21_id1__ already exists)
    ```
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/14752


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14752: [SPARK-17186][SQL] remove catalog table type INDE...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14752#discussion_r76003068
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ---
    @@ -379,8 +379,9 @@ private[hive] class HiveClientImpl(
             tableType = h.getTableType match {
               case HiveTableType.EXTERNAL_TABLE => CatalogTableType.EXTERNAL
               case HiveTableType.MANAGED_TABLE => CatalogTableType.MANAGED
    -          case HiveTableType.INDEX_TABLE => CatalogTableType.INDEX
               case HiveTableType.VIRTUAL_VIEW => CatalogTableType.VIEW
    +          case HiveTableType.INDEX_TABLE =>
    +            throw new AnalysisException("Hive index table is not supported.")
    --- End diff --
    
    This mostly won't happen, but we can improve it by introducing a `isTableExists` API in `HiveClient`, do you wanna work on it? thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64205/consoleFull)** for PR 14752 at commit [`d2bc794`](https://github.com/apache/spark/commit/d2bc794767eef9a357c7cb77cdef017403516cb9).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    **[Test build #64245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64245/consoleFull)** for PR 14752 at commit [`688802b`](https://github.com/apache/spark/commit/688802b7ec40498aba309ebb4ce0a1c667456635).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14752: [SPARK-17186][SQL] remove catalog table type INDEX

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14752
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org