You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/17 05:44:14 UTC
[GitHub] [spark] yaooqinn opened a new pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
yaooqinn opened a new pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
In Spark CLI, we create a hive `CliSessionState` and it does not load the `hive-site.xml`. So the configurations in `hive-site.xml` will not take effects like other spark-hive integration apps.
Also, the warehouse directory is not correctly picked. If the `default` database does not exist, the `CliSessionState` will create one during the first time it talks to the metastore. The `Location` of the default DB will be neither the value of `spark.sql.warehousr.dir` nor the user-specified value of `hive.metastore.warehourse.dir`, but the default value of `hive.metastore.warehourse.dir `which will always be `/user/hive/warehouse`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix bug for Spark SQL cli to pick right confs
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
yes, the non-exists default database will be created in the location specified by the users via `spark.sql.warehouse.dir` or `hive.metastore.warehouse.dir`, or the default value of `spark.sql.warehouse.dir` if none of them specified.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
add cli ut
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914636
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894230
cc @cloud-fan @dongjoon-hyun @maropu, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600120629
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r395360704
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
Review comment:
Hi, All.
This test case broke all `Hive 1.2` profile Jenkins jobs in both `branch-3.0` and `master`.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894556
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393519101
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
##########
@@ -55,44 +54,10 @@ private[sql] class SharedState(
SharedState.setFsUrlStreamHandlerFactory(sparkContext.conf)
- // Load hive-site.xml into hadoopConf and determine the warehouse path we want to use, based on
- // the config from both hive and Spark SQL. Finally set the warehouse config value to sparkConf.
- val warehousePath: String = {
- val configFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
- if (configFile != null) {
- logInfo(s"loading hive config file: $configFile")
- sparkContext.hadoopConfiguration.addResource(configFile)
- }
-
- // hive.metastore.warehouse.dir only stay in hadoopConf
- sparkContext.conf.remove("hive.metastore.warehouse.dir")
- // Set the Hive metastore warehouse path to the one we use
- val hiveWarehouseDir = sparkContext.hadoopConfiguration.get("hive.metastore.warehouse.dir")
- if (hiveWarehouseDir != null && !sparkContext.conf.contains(WAREHOUSE_PATH.key)) {
- // If hive.metastore.warehouse.dir is set and spark.sql.warehouse.dir is not set,
- // we will respect the value of hive.metastore.warehouse.dir.
- sparkContext.conf.set(WAREHOUSE_PATH.key, hiveWarehouseDir)
- logInfo(s"${WAREHOUSE_PATH.key} is not set, but hive.metastore.warehouse.dir " +
- s"is set. Setting ${WAREHOUSE_PATH.key} to the value of " +
- s"hive.metastore.warehouse.dir ('$hiveWarehouseDir').")
- hiveWarehouseDir
- } else {
- // If spark.sql.warehouse.dir is set, we will override hive.metastore.warehouse.dir using
- // the value of spark.sql.warehouse.dir.
- // When neither spark.sql.warehouse.dir nor hive.metastore.warehouse.dir is set,
- // we will set hive.metastore.warehouse.dir to the default value of spark.sql.warehouse.dir.
- val sparkWarehouseDir = sparkContext.conf.get(WAREHOUSE_PATH)
- logInfo(s"Setting hive.metastore.warehouse.dir ('$hiveWarehouseDir') to the value of " +
- s"${WAREHOUSE_PATH.key} ('$sparkWarehouseDir').")
- sparkContext.hadoopConfiguration.set("hive.metastore.warehouse.dir", sparkWarehouseDir)
- sparkWarehouseDir
- }
- }
- logInfo(s"Warehouse path is '$warehousePath'.")
-
- // These 2 variables should be initiated after `warehousePath`, because in the first place we need
Review comment:
ok, I will move it inside.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not
respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890532
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24638/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894565
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24640/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955931
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600085868
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910927
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119913/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393524009
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
I also checked manually
#### now
```sql
kentyao@hulk ~/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317 bin/spark-sql -S --conf spark.sql.warehouse.dir=/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
spark-sql> desc database default;
Database Name default
Comment Default Hive database
Location file:/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
Owner public
```
#### 3.x and 2.4
```sql
kentyao@hulk ~/Downloads/spark/spark-2.4.5-bin-hadoop2.7 bin/spark-sql -S -c spark.sql.warehouse.dir=/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
spark-sql> desc database default;
Database Name default
Description Default Hive database
Location file:/user/hive/warehouse
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914233
**[Test build #119920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119920/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890184
**[Test build #119913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119913/testReport)** for PR 27933 at commit [`13a3fe8`](https://github.com/apache/spark/commit/13a3fe80cbcdde4b73b6829a5421be46e7c022a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894565
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24640/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600392038
cc @juliuszsompolski
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894556
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601456946
When I run `CliSuite` locally with `hive-1.2`, it seems okay weirdly. I'll revert this first and recover 4 Jenkins jobs. Please make a new PR and test with `hive-1.2` tag, @yaooqinn .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393514102
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
Review comment:
why use three `"""` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894279
**[Test build #119915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119915/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914645
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24646/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r395360704
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
Review comment:
Hi, All.
This test case broke all `Hive 1.2` profile Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599973428
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914636
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914233
**[Test build #119920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119920/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601511845
Thanks, @cloud-fan @dongjoon-hyun for the reverting, I will raise a pr to fix this ASAP. Also, sorry for my mistake
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890532
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24638/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600084885
**[Test build #119926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119926/testReport)** for PR 27933 at commit [`51d077e`](https://github.com/apache/spark/commit/51d077e3e7ab30342cbf15b5ad10eb0ca15fd768).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600085886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119926/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910919
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910752
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910919
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599973436
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955489
**[Test build #119926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119926/testReport)** for PR 27933 at commit [`51d077e`](https://github.com/apache/spark/commit/51d077e3e7ab30342cbf15b5ad10eb0ca15fd768).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393524009
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
I also checked manually
#### now
```sql
kentyao@hulk ~/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317 bin/spark-sql -S --conf spark.sql.warehouse.dir=/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
spark-sql> desc database default;
Database Name default
Comment Default Hive database
Location file:/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
Owner public
```
#### 3.x and 2.4
```sql
kentyao@hulk ~/Downloads/spark/spark-2.4.5-bin-hadoop2.7 bin/spark-sql -S -c spark.sql.warehouse.dir=/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-v20200317/data/spark-warehouse
spark-sql> desc formatted default;
Error in query: Table or view 'default' not found in database 'default';
spark-sql> desc database default;
Database Name default
Description Default Hive database
Location file:/user/hive/warehouse
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect
hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890184
**[Test build #119913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119913/testReport)** for PR 27933 at commit [`13a3fe8`](https://github.com/apache/spark/commit/13a3fe80cbcdde4b73b6829a5421be46e7c022a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119915/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599914645
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24646/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910752
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599973428
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393514439
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
how do we verify if the warehouse path is picked?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601511068
Yea let's revert to unblock the builds. @yaooqinn when you open a new PR, please add [hive-1.2] in the title, so jenkins will test with hive 1.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601455586
The situation looks tricky.
```
.../spark-7c73c224-7624-48c5-95dc-a7a8c3bf41bb (Test case is looking for this)
.../spark-e8eb1838-fde8-4465-ada2-86a2b8f07297 (Spark answer in the cluster.)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393514149
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393518456
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
the hive.xxx.xxx.dir is from the global variable `warehousePath`, if the `sparkWareHouseDir` not picked, the db location of default will be the hive one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119915/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599973436
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601455586
The situation looks tricky.
```
.../spark-7c73c224-7624-48c5-95dc-a7a8c3bf41bb (Test case is looking for this)
.../spark-e8eb1838-fde8-4465-ada2-86a2b8f07297 (Spark answer in the Jenkins.)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910667
**[Test build #119913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119913/testReport)** for PR 27933 at commit [`13a3fe8`](https://github.com/apache/spark/commit/13a3fe80cbcdde4b73b6829a5421be46e7c022a9).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955940
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24652/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600085886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119926/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599913165
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601456946
When I run `CliSuite` locally with `hive-1.2`, it seems to okay. I'll revert this first and recover 4 Jenkins jobs. Please make a new PR and test with `hive-1.2` tag, @yaooqinn .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393513526
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
##########
@@ -55,44 +54,10 @@ private[sql] class SharedState(
SharedState.setFsUrlStreamHandlerFactory(sparkContext.conf)
- // Load hive-site.xml into hadoopConf and determine the warehouse path we want to use, based on
- // the config from both hive and Spark SQL. Finally set the warehouse config value to sparkConf.
- val warehousePath: String = {
- val configFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
- if (configFile != null) {
- logInfo(s"loading hive config file: $configFile")
- sparkContext.hadoopConfiguration.addResource(configFile)
- }
-
- // hive.metastore.warehouse.dir only stay in hadoopConf
- sparkContext.conf.remove("hive.metastore.warehouse.dir")
- // Set the Hive metastore warehouse path to the one we use
- val hiveWarehouseDir = sparkContext.hadoopConfiguration.get("hive.metastore.warehouse.dir")
- if (hiveWarehouseDir != null && !sparkContext.conf.contains(WAREHOUSE_PATH.key)) {
- // If hive.metastore.warehouse.dir is set and spark.sql.warehouse.dir is not set,
- // we will respect the value of hive.metastore.warehouse.dir.
- sparkContext.conf.set(WAREHOUSE_PATH.key, hiveWarehouseDir)
- logInfo(s"${WAREHOUSE_PATH.key} is not set, but hive.metastore.warehouse.dir " +
- s"is set. Setting ${WAREHOUSE_PATH.key} to the value of " +
- s"hive.metastore.warehouse.dir ('$hiveWarehouseDir').")
- hiveWarehouseDir
- } else {
- // If spark.sql.warehouse.dir is set, we will override hive.metastore.warehouse.dir using
- // the value of spark.sql.warehouse.dir.
- // When neither spark.sql.warehouse.dir nor hive.metastore.warehouse.dir is set,
- // we will set hive.metastore.warehouse.dir to the default value of spark.sql.warehouse.dir.
- val sparkWarehouseDir = sparkContext.conf.get(WAREHOUSE_PATH)
- logInfo(s"Setting hive.metastore.warehouse.dir ('$hiveWarehouseDir') to the value of " +
- s"${WAREHOUSE_PATH.key} ('$sparkWarehouseDir').")
- sparkContext.hadoopConfiguration.set("hive.metastore.warehouse.dir", sparkWarehouseDir)
- sparkWarehouseDir
- }
- }
- logInfo(s"Warehouse path is '$warehousePath'.")
-
- // These 2 variables should be initiated after `warehousePath`, because in the first place we need
Review comment:
shall we keep this comment? Or we can put `SharedState.loadHiveConfFile` inside `private val (conf, hadoopConf) = {`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393524394
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
How is it verified?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955931
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not
respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890525
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#discussion_r393525824
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##########
@@ -159,6 +160,17 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
}
}
+ test("Pick spark.sql.warehouse.dir first for Spark Cli if set") {
+ val sparkWareHouseDir = Utils.createTempDir()
+ new File(warehousePath, "metastore_db").delete()
+ runCliWithin(
+ 1.minute,
+ Seq("--conf", s"""${StaticSQLConf.WAREHOUSE_PATH.key}=$sparkWareHouseDir"""))(
+ """desc database default;""" -> sparkWareHouseDir.getAbsolutePath
+ )
+ sparkWareHouseDir.delete()
Review comment:
In this test, we will run a spark-sql job to check the output
```
[info] - Pick spark.sql.warehouse.dir first for Spark Cli if set *** FAILED *** (1 minute)
[info] =======================
[info] CliSuite failure output
[info] =======================
[info] Spark SQL CLI command line: ../../bin/spark-sql --master local --driver-java-options -Dderby.system.durability=test --conf spark.ui.enabled=false --hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/Users/kentyao/spark/target/tmp/spark-3570629a-23a7-4e2a-b21a-9bce68025228;create=true --hiveconf hive.metastore.warehouse.dir=/Users/kentyao/spark/target/tmp/spark-dbe4dd6a-1129-43f5-83bc-62dd2072b410 --hiveconf hive.exec.scratchdir=/Users/kentyao/spark/target/tmp/spark-9422968c-94d0-4251-b591-30cbcc061845 --hiveconf conf1=conftest --hiveconf conf2=1 --conf spark.sql.warehouse.dir=/Users/kentyao/spark/target/tmp/spark-1219faf2-918f-4ac4-8d28-fa4dd2d24c7e
[info] Exception: java.util.concurrent.TimeoutException: Futures timed out after [1 minute]
[info] Executed query 0 "desc database default;",
[info] But failed to capture expected output "/Users/kentyao/spark/target/tmp/spark-1219faf2-918f-4ac4-8d28-fa4dd2d24c7e" within 1 minute.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599890525
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955489
**[Test build #119926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119926/testReport)** for PR 27933 at commit [`51d077e`](https://github.com/apache/spark/commit/51d077e3e7ab30342cbf15b5ad10eb0ca15fd768).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-600085868
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599894279
**[Test build #119915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119915/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599955940
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24652/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910927
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119913/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601456946
When I run `CliSuite` locally with `hive-1.2`, it seems okay weirdly. I'll revert this first and recover 4 Jenkins jobs. Please make a new PR and test with `hive-1.2` tag, @yaooqinn . Sorry for the reverting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599973200
**[Test build #119920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119920/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL]
Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601512455
Thank you so much, @cloud-fan and @yaooqinn . The jobs are recovered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27933:
[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-601456946
When I run `CliSuite` locally with `hive-1.2`, it seems okay. I'll revert this first and recover 4 Jenkins jobs. Please make a new PR and test with `hive-1.2` tag, @yaooqinn .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark
SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
URL: https://github.com/apache/spark/pull/27933#issuecomment-599910658
**[Test build #119915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119915/testReport)** for PR 27933 at commit [`665a3ce`](https://github.com/apache/spark/commit/665a3cead0a6431ba9c1bc2550fd2148d8c4f52a).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org