You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/06 06:49:02 UTC
[GitHub] [spark] yaooqinn opened a new pull request #27827: [SPARK-31066]
Disable useless and uncleaned hive SessionState initialization parts
yaooqinn opened a new pull request #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
As a common usage and according to the spark doc, users may often just copy their `hive-site.xml` to Spark directly from hive projects. Sometimes, the config file is not that clean for spark and may cause some side effects.
for example, `hive.session.history.enabled` will create a log for the hive jobs but useless for spark and also it will not be deleted on JVM exit.
this pr 1)disable `hive.session.history.enabled` explicitly to disable creating `hive_job_log` file, e.g.
```
Hive history file=/var/folders/01/h81cs4sn3dq2dd_k4j6fhrmc0000gn/T//kentyao/hive_job_log_79c63b29-95a4-4935-a9eb-2d89844dfe4f_493861201.txt
```
2) set `hive.execution.engine` to `spark` explicitly in case the config is `tez` and casue uneccesary problem like this:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
reduce overhead of internal complexity and users' hive cognitive load for running spark
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
yes, `hive_job_log` file will not be created even enabled, and will not try to initialize tez kinds of stuff
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
add ut and verify manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595697782
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119453/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595652810
**[Test build #119453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119453/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027367
**[Test build #119500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119500/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752894
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027534
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24229/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595652810
**[Test build #119453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119453/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595628033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24183/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912967
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596417628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-600389576
cc @juliuszsompolski
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752572
**[Test build #119466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119466/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-598012351
> Is this fix related only to `hive1.2`? I just want to know which hive version is related to this issue.
No, this mark is not added by me, I have no idea why it is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027367
**[Test build #119500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119500/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912981
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24209/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595653394
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24188/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027200
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595628023
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027530
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595957634
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119479/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912457
**[Test build #119479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119479/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596417638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24293/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595627651
**[Test build #119448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119448/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595827878
**[Test build #119466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119466/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-597501387
gentle ping @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595697599
**[Test build #119453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119453/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596042372
**[Test build #119500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119500/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595828748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119466/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596042486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119500/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595957629
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650676
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595653394
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24188/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595628033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24183/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596420746
**[Test build #119562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119562/testReport)** for PR 27827 at commit [`292b3eb`](https://github.com/apache/spark/commit/292b3eb217c6a559f51ed69fb3dabd955de57332).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595697782
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119453/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595828739
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752894
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596417638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24293/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650542
**[Test build #119448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119448/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596042484
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596420746
**[Test build #119562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119562/testReport)** for PR 27827 at commit [`292b3eb`](https://github.com/apache/spark/commit/292b3eb217c6a559f51ed69fb3dabd955de57332).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-598011925
Is this fix related only to `hive1.2`? I just want to know which hive version is related to this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650676
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596490514
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027530
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119448/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595910399
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752572
**[Test build #119466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119466/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27827:
[SPARK-31066] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r388754145
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
The overwrite is better than just a warning log?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650699
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912981
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24209/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595650680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119448/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752898
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24198/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595828748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119466/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r389541282
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
I have added warnings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r391514642
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -159,36 +160,7 @@ private[hive] class HiveClientImpl(
s"(version ${version.fullVersion}) is ${conf.getVar(ConfVars.METASTOREWAREHOUSE)}")
private def newState(): SessionState = {
- val hiveConf = new HiveConf(classOf[SessionState])
- // HiveConf is a Hadoop Configuration, which has a field of classLoader and
- // the initial value will be the current thread's context class loader
- // (i.e. initClassLoader at here).
- // We call hiveConf.setClassLoader(initClassLoader) at here to make
- // this action explicit.
- hiveConf.setClassLoader(initClassLoader)
-
- // 1: Take all from the hadoopConf to this hiveConf.
- // This hadoopConf contains user settings in Hadoop's core-site.xml file
- // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
- // SharedState and put settings in this hadoopConf instead of relying on HiveConf
- // to load user settings. Otherwise, HiveConf's initialize method will override
- // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
- // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
- // has hive-site.xml. So, HiveConf will use that to override its default values.
- // 2: we set all spark confs to this hiveConf.
- // 3: we set all entries in config to this hiveConf.
- val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
- confMap.foreach { case (k, v) => hiveConf.set(k, v) }
- SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
- logDebug(
- s"""
- |Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
- |$k=$v
- """.stripMargin)
- }
- // Disable CBO because we removed the Calcite dependency.
- hiveConf.setBoolean("hive.cbo.enable", false)
Review comment:
It's not ideal to handle hive-specific configs in Spark. But we already did here, and it makes Spark more user-friendly, I'm OK with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595957401
**[Test build #119479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119479/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912457
**[Test build #119479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119479/testReport)** for PR 27827 at commit [`a52e35f`](https://github.com/apache/spark/commit/a52e35fa382f06a9d93229a2c98a48ac1d2a5f87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595628023
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596490514
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27827: [SPARK-31066] Disable
useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595643254
also cc: @wangyum
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595957634
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119479/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595957629
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596027534
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24229/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596490521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119562/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r389485082
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
Ah, I see. But, is it ok to silently rewrite the config? The overwrite itself ok, but I feel it'd be better to inform users that they should change their config in a next run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596490521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119562/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595752898
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24198/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596042486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119500/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r388771751
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
warning log is not enough, we need reset this before hive `SessionState.start`. And the reason I did not add a warning here is to follow what we did for `hive.cbo.enable`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-598108116
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27827: [SPARK-31066] Disable
useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595626966
cc @cloud-fan @maropu @dongjoon-hyun, thanks for reviewing this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596042484
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595697766
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596417628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595653390
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r389541282
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
I have added warning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27827:
[SPARK-31066][SQL] Disable useless and uncleaned hive SessionState
initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595697766
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-596489810
**[Test build #119562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119562/testReport)** for PR 27827 at commit [`292b3eb`](https://github.com/apache/spark/commit/292b3eb217c6a559f51ed69fb3dabd955de57332).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#discussion_r391406612
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1232,4 +1204,42 @@ private[hive] object HiveClientImpl {
StatsSetupConst.RAW_DATA_SIZE,
StatsSetupConst.TOTAL_SIZE
)
+
+ def newHiveConf(
+ sparkConf: SparkConf,
+ hadoopConf: JIterable[JMap.Entry[String, String]],
+ extraConfig: Map[String, String],
+ classLoader: Option[ClassLoader] = None): HiveConf = {
+ val hiveConf = new HiveConf(classOf[SessionState])
+ // HiveConf is a Hadoop Configuration, which has a field of classLoader and
+ // the initial value will be the current thread's context class loader.
+ // We call hiveConf.setClassLoader(initClassLoader) at here to ensure it use the classloader
+ // we want.
+ classLoader.foreach(hiveConf.setClassLoader)
+ // 1: Take all from the hadoopConf to this hiveConf.
+ // This hadoopConf contains user settings in Hadoop's core-site.xml file
+ // and Hive's hive-site.xml file. Note, we load hive-site.xml file manually in
+ // SharedState and put settings in this hadoopConf instead of relying on HiveConf
+ // to load user settings. Otherwise, HiveConf's initialize method will override
+ // settings in the hadoopConf. This issue only shows up when spark.sql.hive.metastore.jars
+ // is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
+ // has hive-site.xml. So, HiveConf will use that to override its default values.
+ // 2: we set all spark confs to this hiveConf.
+ // 3: we set all entries in config to this hiveConf.
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
+ logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive Conf:$k=$v")
+ }
+ // Disable CBO because we removed the Calcite dependency.
+ hiveConf.setBoolean("hive.cbo.enable", false)
+ // If this is true, SessionState.start will create a file to log hive job which will not be
+ // deleted on exit and is useless for spark
+ hiveConf.setBoolean("hive.session.history.enabled", false)
+ // If this is tez engine, SessionState.start might bring extra logic to initialize tez stuff,
+ // which is useless for spark.
+ hiveConf.set("hive.execution.engine", "spark")
Review comment:
WDYT this fix? @dongjoon-hyun @HyukjinKwon ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827:
[SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive
SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL][test-hive1.2] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595912967
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595653390
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27827: [SPARK-31066] Disable
useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27827: [SPARK-31066] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595627651
**[Test build #119448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119448/testReport)** for PR 27827 at commit [`36c697d`](https://github.com/apache/spark/commit/36c697d394fc11893c25fc009739677fa9b1fa28).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27827: [SPARK-31066][SQL]
Disable useless and uncleaned hive SessionState initialization parts
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27827: [SPARK-31066][SQL] Disable useless and uncleaned hive SessionState initialization parts
URL: https://github.com/apache/spark/pull/27827#issuecomment-595828739
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org