You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/21 14:58:18 UTC

[GitHub] [spark] Ngone51 opened a new pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Ngone51 opened a new pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670
 
 
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   2 migration tips:
   * different string representation of decimal when use `TRANSFORM` syntax
   
   * local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem`
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   Let user be aware of behavior changes when they upgrade to Spark3.0
   
   
   ### Does this PR introduce any user-facing change?
   <!--
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If no, write 'No'.
   -->
   
   No, new doc for Spark 3.0.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   N/A.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887262
 
 
   **[Test build #119024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119024/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887750
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591885817
 
 
   Jenkins, retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591836407
 
 
   **[Test build #119014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119014/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717250
 
 
   **[Test build #118903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118903/testReport)** for PR 27670 at commit [`679497c`](https://github.com/apache/spark/commit/679497ca201cd57684f7bbf1a1bbdb19ab1fb87b).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#discussion_r383074436
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when use Hadoop API, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will returns `true` in Hive 1.2 but `false` in Hive 2.3 if `dst` represents a file.
 
 Review comment:
   > Should we group all of hive related behavior change under the same section or the same bullet point?
   
   ~I'll make it in a separate PR.~
   
   Updated.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590812929
 
 
   **[Test build #118908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118908/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `  - In Spark 3.0, the deprecated `HiveContext` class has been removed. Use `SparkSession.builder.enableHiveSupport()` instead.`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717250
 
 
   **[Test build #118903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118903/testReport)** for PR 27670 at commit [`679497c`](https://github.com/apache/spark/commit/679497ca201cd57684f7bbf1a1bbdb19ab1fb87b).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590179628
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590848254
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590179631
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118840/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887750
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717642
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23652/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156176
 
 
   **[Test build #118838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118838/testReport)** for PR 27670 at commit [`a2a6e1b`](https://github.com/apache/spark/commit/a2a6e1b0244246e02dfddacd9083477b03fbfc90).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r382676613
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
 
 Review comment:
   "The string representation of a decimal ..."
   use -> using
   sql -> SQL
   "In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590848254
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590813543
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590157104
 
 
   **[Test build #118840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118840/testReport)** for PR 27670 at commit [`c310e67`](https://github.com/apache/spark/commit/c310e6790ce2972601beb57ae8bc037df6c5f8e5).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590177281
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590772469
 
 
   **[Test build #118918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118918/testReport)** for PR 27670 at commit [`6c6e7ff`](https://github.com/apache/spark/commit/6c6e7ff5009f2e1a460be313dc2c626c7a9d835d).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736304
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118903/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156333
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23587/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590813556
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118908/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814481
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23761/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736048
 
 
   **[Test build #118904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118904/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `  - In Spark 3.0, the deprecated `HiveContext` class has been removed. Use `SparkSession.builder.enableHiveSupport()` instead.`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156330
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#discussion_r383804597
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -330,10 +330,14 @@ license: |
 
   - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For generating Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead.
 
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3 and it brings following impacts:
+  
+    - This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
   For example: set `spark.sql.hive.metastore.version` to `1.2.1` and `spark.sql.hive.metastore.jars` to `maven` if your Hive metastore version is 1.2.1.
   
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
+    - You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
+
+    - The string representation of a decimal can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` syntax in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary.
 
 Review comment:
   `the string representation`-> `the decimal string representation`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yhuai commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
yhuai commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r382697622
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when use Hadoop API, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will returns `true` in Hive 1.2 but `false` in Hive 2.3 if `dst` represents a file.
 
 Review comment:
   Should we group all of hive related behavior change under the same section or the same bullet point? Then, we make individual changes sub-bullet-points.
   
   Also, when we say rename returns true for hive 1.2 when des exists, we should call out that the destination will get overwritten, right? Also, can we link to related hadoop and hive tickets?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736041
 
 
   **[Test build #118903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118903/testReport)** for PR 27670 at commit [`679497c`](https://github.com/apache/spark/commit/679497ca201cd57684f7bbf1a1bbdb19ab1fb87b).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589690102
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590769414
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814025
 
 
   **[Test build #119014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119014/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590768876
 
 
   **[Test build #118917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118917/testReport)** for PR 27670 at commit [`f5f8928`](https://github.com/apache/spark/commit/f5f892832a62c36f8880a5e4206ae0e995dea32b).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590768876
 
 
   **[Test build #118917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118917/testReport)** for PR 27670 at commit [`f5f8928`](https://github.com/apache/spark/commit/f5f892832a62c36f8880a5e4206ae0e995dea32b).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590769421
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23666/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887759
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23771/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590848260
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118917/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887759
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23771/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814471
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722034
 
 
   **[Test build #118904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118904/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590177281
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589689212
 
 
   cc @cloud-fan @gatorsmile 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590840822
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722429
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23653/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814471
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736289
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#discussion_r383698104
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The string representation of a decimal can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` syntax in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when using Hadoop APIs, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will return `true` and overwrite `dst` in Hive 1.2 but return `false` in Hive 2.3 if `dst` is an existed file. See HIVE-13705 and HDFS-10385 for more details.
 
 Review comment:
   Removed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591954464
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r383074436
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when use Hadoop API, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will returns `true` in Hive 1.2 but `false` in Hive 2.3 if `dst` represents a file.
 
 Review comment:
   > Should we group all of hive related behavior change under the same section or the same bullet point?
   
   I'll make it in a separate PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814481
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23761/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722421
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590840822
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736142
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#discussion_r383752735
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -252,7 +252,7 @@ license: |
         </tr>
     </table>
     
-  - Since Spark 3.0, CREATE TABLE without a specific provider will use the value of `spark.sql.sources.default` as its provider. In Spark version 2.4 and earlier, it was hive. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.createHiveTableByDefault.enabled` to `true`.
+  - Since Spark 3.0, `CREATE TABLE` without a specific provider will use the value of `spark.sql.sources.default` as its provider. In Spark version 2.4 and earlier, it was hive. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.createHiveTableByDefault.enabled` to `true`.
 
 Review comment:
   minor fix: CREATE TABLE -> 'CREATE TABLE'

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589759715
 
 
   **[Test build #118796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118796/testReport)** for PR 27670 at commit [`5474b74`](https://github.com/apache/spark/commit/5474b749518d64eb8bfac2e4efc3ff7dd979d0ac).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590177286
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118838/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717634
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156333
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23587/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590179628
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736812
 
 
   retest this please.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722034
 
 
   **[Test build #118904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118904/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589760343
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589690114
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23547/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591954464
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590738356
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23657/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589760343
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-592257284
 
 
   thanks all!!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590737876
 
 
   **[Test build #118908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118908/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591836507
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119014/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722429
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23653/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156176
 
 
   **[Test build #118838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118838/testReport)** for PR 27670 at commit [`a2a6e1b`](https://github.com/apache/spark/commit/a2a6e1b0244246e02dfddacd9083477b03fbfc90).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590813556
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118908/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589760352
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118796/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Ngone51 commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156051
 
 
   Thanks for the review, updated. @srowen @yhuai 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736142
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yhuai commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
yhuai commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r382696919
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
 
 Review comment:
   Let's call it out transform is used for script transformation, which depends on hive's behavior. 
   
   I was not clear how a hive change can affect spark. But they realized that transform is script transformation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590813543
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591836507
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119014/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590179214
 
 
   **[Test build #118840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118840/testReport)** for PR 27670 at commit [`c310e67`](https://github.com/apache/spark/commit/c310e6790ce2972601beb57ae8bc037df6c5f8e5).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r383088090
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The string representation of a decimal can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` syntax in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when using Hadoop APIs, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will return `true` and overwrite `dst` in Hive 1.2 but return `false` in Hive 2.3 if `dst` is an existed file. See HIVE-13705 and HDFS-10385 for more details.
 
 Review comment:
   Yea, we should highlight the changes for end-users, not internal differences. If we can't find one, let's remove this item.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r383087473
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The string representation of a decimal can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` syntax in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when using Hadoop APIs, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will return `true` and overwrite `dst` in Hive 1.2 but return `false` in Hive 2.3 if `dst` is an existed file. See HIVE-13705 and HDFS-10385 for more details.
 
 Review comment:
   How does this relate to Spark's behaviour?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591954471
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119024/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590737876
 
 
   **[Test build #118908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118908/testReport)** for PR 27670 at commit [`0910a4c`](https://github.com/apache/spark/commit/0910a4c0192a8f724c64898a8cd55592d6e62135).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590177286
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118838/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589760352
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118796/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717634
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589689511
 
 
   **[Test build #118796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118796/testReport)** for PR 27670 at commit [`5474b74`](https://github.com/apache/spark/commit/5474b749518d64eb8bfac2e4efc3ff7dd979d0ac).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#discussion_r383803345
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -330,10 +330,14 @@ license: |
 
   - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For generating Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead.
 
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3 and it brings following impacts:
+  
+    - This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
 
 Review comment:
   `This may need` -> `You may need`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736154
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118904/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590840837
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118918/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591953761
 
 
   **[Test build #119024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119024/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590769421
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23666/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736154
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118904/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590179631
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118840/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590738356
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23657/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736289
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591954471
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119024/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590738341
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#discussion_r382676896
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -335,6 +335,10 @@ license: |
   
   - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
 
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The result decimal represent in string can be different between Hive 1.2 and Hive 2.3 when use `TRANSFORM` syntax in sql. In Hive 1.2, decimal will eliminate all non-significant zeros in precision when represents in string. But in Hive 2.3, it will always show 18 digits in precision and make up with zero if its precision is less than 18.
+  
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. The local file system has changed from `org.apache.hadoop.fs.LocalFileSystem` to `org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem` when use Hadoop API, e.g. `new Path("file:/").getFileSystem(conf)`, and `fs.rename(Path src, Path dst)` will returns `true` in Hive 1.2 but `false` in Hive 2.3 if `dst` represents a file.
 
 Review comment:
   use -> using
   API -> APIs
   returns -> return

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591836494
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591814025
 
 
   **[Test build #119014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119014/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590736304
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118903/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#discussion_r383805108
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -330,10 +330,14 @@ license: |
 
   - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For generating Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead.
 
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3 and it brings following impacts:
+  
+    - This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
   For example: set `spark.sql.hive.metastore.version` to `1.2.1` and `spark.sql.hive.metastore.jars` to `maven` if your Hive metastore version is 1.2.1.
   
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
+    - You need to migrate your custom SerDes to Hive 2.3 or build your own Spark with `hive-1.2` profile. See HIVE-15167 for more details.
+
+    - The string representation of a decimal can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` syntax in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary.
 
 Review comment:
   TRANSFORM syntax -> TRANSFORM operator

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590177017
 
 
   **[Test build #118838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118838/testReport)** for PR 27670 at commit [`a2a6e1b`](https://github.com/apache/spark/commit/a2a6e1b0244246e02dfddacd9083477b03fbfc90).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591971101
 
 
   thanks, merging to master/3.0!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590717642
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23652/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590157104
 
 
   **[Test build #118840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118840/testReport)** for PR 27670 at commit [`c310e67`](https://github.com/apache/spark/commit/c310e6790ce2972601beb57ae8bc037df6c5f8e5).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589690114
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23547/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589689511
 
 
   **[Test build #118796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118796/testReport)** for PR 27670 at commit [`5474b74`](https://github.com/apache/spark/commit/5474b749518d64eb8bfac2e4efc3ff7dd979d0ac).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-589690102
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590738341
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590840062
 
 
   **[Test build #118918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118918/testReport)** for PR 27670 at commit [`6c6e7ff`](https://github.com/apache/spark/commit/6c6e7ff5009f2e1a460be313dc2c626c7a9d835d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590772469
 
 
   **[Test build #118918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118918/testReport)** for PR 27670 at commit [`6c6e7ff`](https://github.com/apache/spark/commit/6c6e7ff5009f2e1a460be313dc2c626c7a9d835d).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [MINOR][DOC] Add two migration tips for Hive
URL: https://github.com/apache/spark/pull/27670#issuecomment-590156330
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591836494
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590848260
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118917/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590847681
 
 
   **[Test build #118917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118917/testReport)** for PR 27670 at commit [`f5f8928`](https://github.com/apache/spark/commit/f5f892832a62c36f8880a5e4206ae0e995dea32b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590840837
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118918/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-591887262
 
 
   **[Test build #119024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119024/testReport)** for PR 27670 at commit [`b7e04b7`](https://github.com/apache/spark/commit/b7e04b7b52b9ff3c14f0863b65f090b973dc4e86).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#discussion_r383803684
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -330,10 +330,14 @@ license: |
 
   - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For generating Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead.
 
-  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3. This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
+  - Since Spark 3.0, we upgraded the built-in Hive from 1.2 to 2.3 and it brings following impacts:
+  
+    - This may need to set `spark.sql.hive.metastore.version` and `spark.sql.hive.metastore.jars` according to the version of the Hive metastore.
 
 Review comment:
   `... of the Hive metastore you want to connect to`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27670: [SPARK-30937][DOC] Group Hive upgrade guides together
URL: https://github.com/apache/spark/pull/27670#issuecomment-590769414
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27670: [SPARK-30937][DOC] Group Hive migration guides into a sub section of Spark3.0
URL: https://github.com/apache/spark/pull/27670#issuecomment-590722421
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org