You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by xwu0226 <gi...@git.apache.org> on 2016/06/15 19:00:50 UTC

[GitHub] spark pull request #13687: [SPARK-15970][SQL]: avoid warning message for hiv...

GitHub user xwu0226 opened a pull request:

    https://github.com/apache/spark/pull/13687

    [SPARK-15970][SQL]: avoid warning message for hive metastore in In-memory ca…

    ## What changes were proposed in this pull request?
    ##### Issue:
    Switch the catalog implementation mode to in-memory mode by adding `spark.sql.catalogImplementation                 in-memory` in `spark-defaults.conf`. 
    Then, start spark-shell, which will be run using in-memory catalog.
    Issue DDL command like `CREATE TABLE T1 (c1 INT, c2 INT) USING PARQUET PARTITIONED BY (c2)`. The table metadata will be created in in-memory catalog. However, a WARNING message dumped into the console saying that `Persisting partitioned data source relation default.t1 into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.`  This kind of warning message is not applicable to In-Memory catalog mode.
    
    This PR is to skip checking whether the table can be created as hive compatible table or not when in-memory catalog mode is in use.
    
    ## How was this patch tested?
    A unit test case is added for in-memory catalog mode. And regrets is run locally.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xwu0226/spark SPARK-15970

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13687
    
----
commit f6b17174fb3896af002d2d2f81bea117f49e7b57
Author: Xin Wu <xi...@us.ibm.com>
Date:   2016-06-15T18:49:45Z

    SPARK-15970: avoid warning message for hive metastore in In-memory catalog mode

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    **[Test build #62726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62726/consoleFull)** for PR 13687 at commit [`f6b1717`](https://github.com/apache/spark/commit/f6b17174fb3896af002d2d2f81bea117f49e7b57).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    **[Test build #69748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69748/consoleFull)** for PR 13687 at commit [`f6b1717`](https://github.com/apache/spark/commit/f6b17174fb3896af002d2d2f81bea117f49e7b57).
     * This patch passes all tests.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    **[Test build #60588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60588/consoleFull)** for PR 13687 at commit [`f6b1717`](https://github.com/apache/spark/commit/f6b17174fb3896af002d2d2f81bea117f49e7b57).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    **[Test build #62726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62726/consoleFull)** for PR 13687 at commit [`f6b1717`](https://github.com/apache/spark/commit/f6b17174fb3896af002d2d2f81bea117f49e7b57).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62726/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60588/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69748/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by xwu0226 <gi...@git.apache.org>.

Github user xwu0226 commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Closed since the hive refactoring code has resolve this issue by the way. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13687: [SPARK-15970][SQL]: avoid warning message for hive metas...

Posted by xwu0226 <gi...@git.apache.org>.

Github user xwu0226 commented on the issue:

    https://github.com/apache/spark/pull/13687
  
    Let me rebase and code and try again . Will close it if we don't have any issue. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org