You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/13 06:04:42 UTC
[GitHub] [spark] yaooqinn opened a new pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
yaooqinn opened a new pull request #32144:
URL: https://github.com/apache/spark/pull/32144
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Currently, pure SQL users are short of ways to see the Hadoop configurations which may affect their jobs a lot
In this PR, I propose to support get a Hadoop configuration by SET syntax, for example
```
SET mapreduce.map.output.compress.codec;
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
better use experience for pure SQL users
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
yes, where retrieving a conf only existing in sessionState.hadoopConf, before is `undefined` and now you see it
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
new test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819321315
**[Test build #137321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137321/testReport)** for PR 32144 at commit [`e612eab`](https://github.com/apache/spark/commit/e612eab3ab897e8629d1ce20921927f75b2b738a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818785154
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612434588
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,13 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("Support retrieve hadoop configuration via set command") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ spark.sharedState.hadoopConf.set(key, value)
Review comment:
OK, I will add check for keys in `src/test/resources/hive-site.xml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819460289
**[Test build #137348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137348/testReport)** for PR 32144 at commit [`8dbed7b`](https://github.com/apache/spark/commit/8dbed7be1f06d6ad36abd73d88a4a1ce4c5ae515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818538647
I respectfully disagree. The problem is that you can have multiple tenants on the same cluster. People will become very unhappy when you allow users to change global configurations. If you want this done, you will need to introduce hadoop configurations at the session level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819490602
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818524464
I am not entirely sure that using SET syntax for reading a value is a good idea. People will expect that you can also modify them, which IMO should not be supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612422398
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,13 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("Support retrieve hadoop configuration via set command") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ spark.sharedState.hadoopConf.set(key, value)
Review comment:
Can we test with a valid use case? This doesn't seem to be valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819484529
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819451609
**[Test build #137346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137346/testReport)** for PR 32144 at commit [`23248c3`](https://github.com/apache/spark/commit/23248c32afda00b00f41cdfbfd3e063fc10a41c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819570206
**[Test build #137351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137351/testReport)** for PR 32144 at commit [`a3a6c5e`](https://github.com/apache/spark/commit/a3a6c5e8401b6af21c329d71928a353432142349).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819793496
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137353/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r616839583
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,21 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("SPARK-35044: SET command shall display default value for hadoop conf correctly") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ // these keys are located at `src/test/resources/hive-site.xml`
+ checkAnswer(sql(s"SET $key"), Row(key, value))
+ checkAnswer(sql("SET hadoop.tmp.dir"), Row("hadoop.tmp.dir", "/tmp/hive_one"))
+
+ // these keys does not exist as default yet
+ checkAnswer(sql(s"SET ${key}no"), Row(key + "no", "<undefined>"))
+ checkAnswer(sql("SET dfs.replication"), Row("dfs.replication", "<undefined>"))
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819221116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41901/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612465244
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
##########
@@ -146,7 +146,9 @@ case class SetCommand(kv: Option[(String, Option[String])])
// Queries a single property.
case Some((key, None)) =>
val runFunc = (sparkSession: SparkSession) => {
- val value = sparkSession.conf.getOption(key).getOrElse("<undefined>")
+ val value = sparkSession.conf.getOption(key).getOrElse {
+ sparkSession.sharedState.hadoopConf.get(key, "<undefined>")
Review comment:
let's add a few comments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612393515
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
##########
@@ -146,7 +146,9 @@ case class SetCommand(kv: Option[(String, Option[String])])
// Queries a single property.
case Some((key, None)) =>
val runFunc = (sparkSession: SparkSession) => {
- val value = sparkSession.conf.getOption(key).getOrElse("<undefined>")
+ val value = sparkSession.conf.getOption(key).getOrElse {
+ sparkSession.sharedState.hadoopConf.get(key, "<undefined>")
Review comment:
> Can we make the PR title clearer?
OK
> BTW do we have a valid use case? e.g. a hadoop conf is not in `SQLConf` but in `sparkSession.sharedState.hadoopConf`.
The pre-loaded ones from `core-site.xml, hive-site.xml` etc., will only stay in `sparkSession.sharedState.hadoopConf` or `sc. _hadoopConfiguation` not `SQLConf`. Some of them that related the Hive Metastore connection(never change it spark runtime), e.g. `hive.metastore.uris`, are clearly global static and unchangeable but displayable I guess. Some of the ones that might be related to, for example, the output codec/compression, preset in Hadoop/hive config files like `core-site.xml` shall bestill changeable from case to case, table to table, file to file, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612393515
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
##########
@@ -146,7 +146,9 @@ case class SetCommand(kv: Option[(String, Option[String])])
// Queries a single property.
case Some((key, None)) =>
val runFunc = (sparkSession: SparkSession) => {
- val value = sparkSession.conf.getOption(key).getOrElse("<undefined>")
+ val value = sparkSession.conf.getOption(key).getOrElse {
+ sparkSession.sharedState.hadoopConf.get(key, "<undefined>")
Review comment:
> Can we make the PR title clearer?
OK
> BTW do we have a valid use case? e.g. a hadoop conf is not in `SQLConf` but in `sparkSession.sharedState.hadoopConf`.
The pre-loaded ones from `core-site.xml, hive-site.xml` etc., will only stay in `sparkSession.sharedState.hadoopConf` or `sc. _hadoopConfiguation` not `SQLConf`. Some of them that related the Hive Metastore connection(never change it spark runtime), e.g. `hive.metastore.uris`, are clearly global static and unchangeable but displayable I guess. Some of the ones that might be related to, for example, the output codec/compression, preset in Hadoop/hive config files like `core-site.xml` shall bestill changeable from case to case, table to table, file to file, etc. So, it's meaningful to show the defaults for users to change based on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819575442
**[Test build #137353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137353/testReport)** for PR 32144 at commit [`b6d26b7`](https://github.com/apache/spark/commit/b6d26b7d87917e7e230ca38f2817c6110b461370).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819451609
**[Test build #137346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137346/testReport)** for PR 32144 at commit [`23248c3`](https://github.com/apache/spark/commit/23248c32afda00b00f41cdfbfd3e063fc10a41c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818533799
> People will expect that you can also modify them, which IMO should not be supported.
We currently do need to support some Hive/Hadoop configurations at runtime, (for example, the STS), such as turning off/on the compression via `mapreduce.map.output.compress.codec`/ `hive.exec.compress.output`, which is modifiable but not able to see the default value. Returning the `<undefined>` causes ambiguity as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818828974
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819684782
**[Test build #749140422](https://github.com/yaooqinn/spark/actions/runs/749140422)** for PR 32144 at commit [`8dbed7b`](https://github.com/yaooqinn/spark/commit/8dbed7be1f06d6ad36abd73d88a4a1ce4c5ae515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819791485
**[Test build #137353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137353/testReport)** for PR 32144 at commit [`b6d26b7`](https://github.com/apache/spark/commit/b6d26b7d87917e7e230ca38f2817c6110b461370).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818533799
> People will expect that you can also modify them, which IMO should not be supported.
We currently do need to support some Hive/Hadoop configurations at runtime, (for example, the STS), such as turning off/on the compression via `mapreduce.map.output.compress.codec`, which is modifiable but not able to see the default value. Returning the `<undefined>` causes ambiguity as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818578354
also cc @cloud-fan @maropu @wangyum @HyukjinKwon and thanks very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818512662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819570206
**[Test build #137351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137351/testReport)** for PR 32144 at commit [`a3a6c5e`](https://github.com/apache/spark/commit/a3a6c5e8401b6af21c329d71928a353432142349).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818780876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819658081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818550752
> If you want this done, you will need to introduce hadoop configurations at the session level.
The current behavior of the PR is equivalent to `sparkSession.sessionState.newHadoopConf().get(key)`, I guess this meets your point of ` the session level`.
```scala
private[sql] object SessionState {
def newHadoopConf(hadoopConf: Configuration, sqlConf: SQLConf): Configuration = {
val newHadoopConf = new Configuration(hadoopConf)
sqlConf.getAllConfs.foreach { case (k, v) => if (v ne null) newHadoopConf.set(k, v) }
newHadoopConf
}
}
```
Using the global `sharedState.hadoopConf` here is only to avoid the unneccary copy.
As I don't change the write-side, I believe it's also at the session level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818831393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819658081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819579649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819216315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r613153508
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,21 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("SPARK-35044: SET command shall display default value for hadoop conf correctly") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ // these keys are located at `src/test/resources/hive-site.xml`
+ checkAnswer(sql(s"SET $key"), Row(key, value))
+ checkAnswer(sql("SET hadoop.tmp.dir"), Row("hadoop.tmp.dir", "/tmp/hive_one"))
+
+ // this key does not exist
+ checkAnswer(sql(s"SET ${key}no"), Row(key + "no", "<undefined>"))
+ checkAnswer(sql("SET dfs.replication"), Row("dfs.replication", "<undefined>"))
+
+ // io.file.buffer.size has a default value from `SparkHadoopUtil.newConfiguration`
+ checkAnswer(sql("SET io.file.buffer.size"), Row("io.file.buffer.size", "65536"))
Review comment:
w/o the patch
```scala
spark-sql> SET io.file.buffer.size;
io.file.buffer.size <undefined>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818669651
**[Test build #137277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137277/testReport)** for PR 32144 at commit [`ab79682`](https://github.com/apache/spark/commit/ab796829f88fe6ed852901c12f40c762bbb7156c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819490566
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818831393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819631255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819340775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819614915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41930/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819610409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn closed pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #32144:
URL: https://github.com/apache/spark/pull/32144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819460289
**[Test build #137348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137348/testReport)** for PR 32144 at commit [`8dbed7b`](https://github.com/apache/spark/commit/8dbed7be1f06d6ad36abd73d88a4a1ce4c5ae515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819578976
**[Test build #137348 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137348/testReport)** for PR 32144 at commit [`8dbed7b`](https://github.com/apache/spark/commit/8dbed7be1f06d6ad36abd73d88a4a1ce4c5ae515).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819205181
**[Test build #137321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137321/testReport)** for PR 32144 at commit [`e612eab`](https://github.com/apache/spark/commit/e612eab3ab897e8629d1ce20921927f75b2b738a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612466239
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,15 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("SPARK-35044: `SET propertyKey` shall lookup sharedState.hadoopConf to display the" +
Review comment:
nit: `SPARK-35044: SET command shall display default value for hadoop conf correctly`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612374123
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
##########
@@ -146,7 +146,9 @@ case class SetCommand(kv: Option[(String, Option[String])])
// Queries a single property.
case Some((key, None)) =>
val runFunc = (sparkSession: SparkSession) => {
- val value = sparkSession.conf.getOption(key).getOrElse("<undefined>")
+ val value = sparkSession.conf.getOption(key).getOrElse {
+ sparkSession.sharedState.hadoopConf.get(key, "<undefined>")
Review comment:
Can we make the PR title clearer? It seems the SET command can already set/get hadoop configs in the SQLConf. The only problem is we don't display the default value correctly, which is from `sparkSession.sharedState.hadoopConf`.
BTW do we have a valid use case? e.g. a hadoop conf is not in `SQLConf` but in `sparkSession.sharedState.hadoopConf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818479656
**[Test build #137277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137277/testReport)** for PR 32144 at commit [`ab79682`](https://github.com/apache/spark/commit/ab796829f88fe6ed852901c12f40c762bbb7156c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819690465
**[Test build #749140422](https://github.com/yaooqinn/spark/actions/runs/749140422)** for PR 32144 at commit [`a3a6c5e`](https://github.com/yaooqinn/spark/commit/a3a6c5e8401b6af21c329d71928a353432142349).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818511702
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819793557
**[Test build #137351 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137351/testReport)** for PR 32144 at commit [`a3a6c5e`](https://github.com/apache/spark/commit/a3a6c5e8401b6af21c329d71928a353432142349).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819490602
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819816072
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137351/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819481264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818785154
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818550752
> If you want this done, you will need to introduce hadoop configurations at the session level.
The current behavior of the PR is equivalent to `sparkSession.sessionState.newHadoopConf().get(key)`, I guess this meets your point of ` the session level`.
```scala
private[sql] object SessionState {
def newHadoopConf(hadoopConf: Configuration, sqlConf: SQLConf): Configuration = {
val newHadoopConf = new Configuration(hadoopConf)
sqlConf.getAllConfs.foreach { case (k, v) => if (v ne null) newHadoopConf.set(k, v) }
newHadoopConf
}
}
```
Using the global `sharedState.hadoopConf` here is only to avoid the unnecessary copy.
As I don't change the write-side, I believe it's also at the session level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818564517
> People will become very unhappy when you allow users to change global configurations.
The changes made are not global. As a system administrator of the computing layer(e.g. me), personally, I am happy that the end-users can solve their problems case by case with our suggestions and dynamically verify and change the relative hive/hadoop configs at the session-level. On the contract, we will probably be unhappy when we need to change it globally at the server-side, which is very likely to affect other users' jobs and cause reboot of a resident service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819614915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41930/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819484529
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819691416
**[Test build #749140422](https://github.com/yaooqinn/spark/actions/runs/749140422)** for PR 32144 at commit [`b6d26b7`](https://github.com/yaooqinn/spark/commit/b6d26b7d87917e7e230ca38f2817c6110b461370).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819679970
**[Test build #749140422](https://github.com/yaooqinn/spark/actions/runs/749140422)** for PR 32144 at commit [`23248c3`](https://github.com/yaooqinn/spark/commit/23248c32afda00b00f41cdfbfd3e063fc10a41c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819575442
**[Test build #137353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137353/testReport)** for PR 32144 at commit [`b6d26b7`](https://github.com/apache/spark/commit/b6d26b7d87917e7e230ca38f2817c6110b461370).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819487605
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819205181
**[Test build #137321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137321/testReport)** for PR 32144 at commit [`e612eab`](https://github.com/apache/spark/commit/e612eab3ab897e8629d1ce20921927f75b2b738a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819579649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819221116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41901/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818564517
> People will become very unhappy when you allow users to change global configurations.
The changes made are not global. As a system administrator of the computing layer(e.g. me), personally, I am happy that the end-users can solve their problems case by case with our suggestions and dynamically verify and change the relative hive/hadoop configs at the session-level. On the contract, we will probably be unhappy when we need to change it globally at the server-side, which is very likely to affect other users' jobs and cause the reboot of resident service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r616833099
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,21 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("SPARK-35044: SET command shall display default value for hadoop conf correctly") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ // these keys are located at `src/test/resources/hive-site.xml`
+ checkAnswer(sql(s"SET $key"), Row(key, value))
+ checkAnswer(sql("SET hadoop.tmp.dir"), Row("hadoop.tmp.dir", "/tmp/hive_one"))
+
+ // these keys does not exist as default yet
+ checkAnswer(sql(s"SET ${key}no"), Row(key + "no", "<undefined>"))
+ checkAnswer(sql("SET dfs.replication"), Row("dfs.replication", "<undefined>"))
Review comment:
This breaks the Hadoop 2.7 tests, seems the behavior is different between hadoop 2.7 and hadoop 3: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.7/2364/testReport/org.apache.spark.sql/SQLQuerySuite/SPARK_35044__SET_command_shall_display_default_value_for_hadoop_conf_correctly/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818742077
**[Test build #137291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137291/testReport)** for PR 32144 at commit [`15f1f74`](https://github.com/apache/spark/commit/15f1f74f8c5ef2ad5adef182a4bed71fedd62a75).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r616837647
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -1069,6 +1069,21 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
spark.sessionState.conf.clear()
}
+ test("SPARK-35044: SET command shall display default value for hadoop conf correctly") {
+ val key = "hadoop.this.is.a.test.key"
+ val value = "2018-11-17 13:33:33.333"
+ // these keys are located at `src/test/resources/hive-site.xml`
+ checkAnswer(sql(s"SET $key"), Row(key, value))
+ checkAnswer(sql("SET hadoop.tmp.dir"), Row("hadoop.tmp.dir", "/tmp/hive_one"))
+
+ // these keys does not exist as default yet
+ checkAnswer(sql(s"SET ${key}no"), Row(key + "no", "<undefined>"))
+ checkAnswer(sql("SET dfs.replication"), Row("dfs.replication", "<undefined>"))
Review comment:
I picked a different config which seems to work: `dfs.hosts`
@yaooqinn can you create a followup PR to try it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819705470
Jenkins passed for many empty commits the GA also passed before rebasing master, the `fast forward` is not related
Thanks for reviewing and merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818671282
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818479656
**[Test build #137277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137277/testReport)** for PR 32144 at commit [`ab79682`](https://github.com/apache/spark/commit/ab796829f88fe6ed852901c12f40c762bbb7156c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818671282
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819340775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818787444
**[Test build #137293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137293/testReport)** for PR 32144 at commit [`d26fbbb`](https://github.com/apache/spark/commit/d26fbbb644ef8357724b037afdafe3d24c14bf21).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] Support retrieve hadoop configurations via SET syntax
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-818512662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819793496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32144:
URL: https://github.com/apache/spark/pull/32144#issuecomment-819647368
**[Test build #137346 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137346/testReport)** for PR 32144 at commit [`23248c3`](https://github.com/apache/spark/commit/23248c32afda00b00f41cdfbfd3e063fc10a41c5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org