You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/10/14 18:51:17 UTC

[GitHub] spark pull request #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.co...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/19499

    [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMetastoreOrc by default

    ## What changes were proposed in this pull request?
    
    Like Parquet, this PR aims to turn on `spark.sql.hive.convertMetastoreOrc` by default.
    
    ## How was this patch tested?
    
    Pass all the existing test cases.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-22279

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19499
    
----
commit b9c495490ca5b3ce07b413f9c4cc7b2f2e1d713b
Author: Dongjoon Hyun <do...@apache.org>
Date:   2017-10-14T18:49:27Z

    [SPARK-22279][SQL] Turn on spark.sql.hive.convertMetastoreOrc by default

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convertMetasto...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19499
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84604/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19499
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convert...

Posted by gatorsmile <gi...@git.apache.org>.

Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19499#discussion_r155397898
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
    @@ -106,7 +106,7 @@ private[spark] object HiveUtils extends Logging {
         .doc("When set to true, the built-in ORC reader and writer are used to process " +
           "ORC tables created by using the HiveQL syntax, instead of Hive serde.")
         .booleanConf
    -    .createWithDefault(false)
    +    .createWithDefault(true)
    --- End diff --
    
    This change was made in https://issues.apache.org/jira/browse/SPARK-15705. 
    
    This issue has been resolved?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convert...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19499#discussion_r155398624
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
    @@ -106,7 +106,7 @@ private[spark] object HiveUtils extends Logging {
         .doc("When set to true, the built-in ORC reader and writer are used to process " +
           "ORC tables created by using the HiveQL syntax, instead of Hive serde.")
         .booleanConf
    -    .createWithDefault(false)
    +    .createWithDefault(true)
    --- End diff --
    
    Yes, it's resolved as you see the last my comment on the JIRA.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19499
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82765/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19499
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82824/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convert...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19499#discussion_r155403790
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
    @@ -106,7 +106,7 @@ private[spark] object HiveUtils extends Logging {
         .doc("When set to true, the built-in ORC reader and writer are used to process " +
           "ORC tables created by using the HiveQL syntax, instead of Hive serde.")
         .booleanConf
    -    .createWithDefault(false)
    +    .createWithDefault(true)
    --- End diff --
    
    Yep. It's resolved via SPARK-14387. The following is a result from 2.2.1.
    ```sql
    hive> CREATE TABLE source(inv_date_sk INT, inv_item_sk INT, inv_warehouse_sk INT, inv_quantity_on_hand INT);
    hive> INSERT INTO source VALUES(1,1,1,1);
    hive> CREATE TABLE inventory(inv_date_sk INT, inv_item_sk INT, inv_warehouse_sk INT, inv_quantity_on_hand INT)
        > ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS ORC;
    hive> INSERT OVERWRITE TABLE inventory SELECT * FROM source;
    
    scala> sql("set spark.sql.hive.convertMetastoreOrc=true")
    scala> sql("SELECT * FROM inventory").show
    +-----------+-----------+----------------+--------------------+
    |inv_date_sk|inv_item_sk|inv_warehouse_sk|inv_quantity_on_hand|
    +-----------+-----------+----------------+--------------------+
    |          1|          1|               1|                   1|
    +-----------+-----------+----------------+--------------------+
    scala> spark.version
    res2: String = 2.2.1
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org