You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/29 10:31:57 UTC
[GitHub] [spark] ajithme opened a new pull request #27041: [SPARK-25061]
Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
ajithme opened a new pull request #27041: [SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
URL: https://github.com/apache/spark/pull/27041
As per the documentation here, https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html user can provide `--hiveconf` to override the hive configurations when using `start-thriftserver.sh` but as per the code, https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L182 here, the hive-site properties (part of hadoopConf) will override the configuration done from command line which is not as per expectation
### What changes were proposed in this pull request?
`hiveConf` need to take precedence when creating `HiveConf`
### Why are the changes needed?
Precedence to `--hiveconf` over `hive-site.xml`
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Tested this patch manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573883221
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587581937
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118644/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme edited a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme edited a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574224827
@dongjoon-hyun I have updated the PR with testcase failure correction also added a UT to reproduce and verify the issue. Please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yhuai commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
yhuai commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r366528640
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,8 +177,12 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
+ sparkConf.getAll.toMap ++ overriddenHiveProps ++ extraConfig).toMap
Review comment:
Seems we should update https://github.com/apache/spark/pull/27041/files#diff-6fd847124f8eae45ba2de1cf7d6296feR170-R179 and also explain why extraConfig is at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377825
**[Test build #116541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116541/testReport)** for PR 27041 at commit [`6969ec2`](https://github.com/apache/spark/commit/6969ec290fc6981aeb39a3eeb2f0d8d878ef4b9e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573385210
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061] Spark SQL
Thrift Server fails to not pick up hiveconf passing parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569493314
cc @srowen @dongjoon-hyun @HeartSaVioR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573828395
Found the test case failures due to `org.apache.spark.sql.hive.client.HiveClientImpl#extraConfig` sent by `org.apache.spark.sql.hive.HiveUtils#newTemporaryConfiguration` were lost due to `overriddenHiveProps`. Hence fixing the order i.e
`hadoopConf < sparkConf < overrideProps< extraConfig`
Please retest. Will update the PR description with manual steps for verification shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing
parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569493029
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507816
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23397/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573883231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116656/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yhuai commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
yhuai commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r366542261
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,8 +177,12 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
+ sparkConf.getAll.toMap ++ overriddenHiveProps ++ extraConfig).toMap
Review comment:
Thank you. As getConfSystemProperties will get all of hive confs that are in the system properties, it is possible that we will pull in a config that is not set by `--hiveconf`. Seems we are introducing a behavior change? Can you explain the impact of this change and why this change is fine?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573385211
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116541/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339421
**[Test build #116725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116725/testReport)** for PR 27041 at commit [`2ee9f61`](https://github.com/apache/spark/commit/2ee9f617b88bef4cd6ba5b2b4c293837bd3dc6bb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573827546
**[Test build #116656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116656/testReport)** for PR 27041 at commit [`cd704bd`](https://github.com/apache/spark/commit/cd704bdd87b2c978ee0ebc76d3a2a9aeb0092453).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339421
**[Test build #116725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116725/testReport)** for PR 27041 at commit [`2ee9f61`](https://github.com/apache/spark/commit/2ee9f617b88bef4cd6ba5b2b4c293837bd3dc6bb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061] Precedence
for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-570187234
gentle ping @HyukjinKwon @steveloughran @cloud-fan @yhuai @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r380464645
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -176,9 +176,18 @@ private[hive] class HiveClientImpl(
// is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
- // 3: we set all entries in config to this hiveConf.
+ // 3: we take the conf passed as --hiveconf which would be set as system properties
+ // by org.apache.hive.service.server.HiveServer2.ServerOptionsProcessor.parse in
+ // org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main.
+ // 4: we set all entries in extraConfig to this hiveConf which have the highest precedence.
+ // To summarize, the order of precedence will be
+ // hadoopConf < sparkConf < overrideProps < extraConfig
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
Review comment:
Can you add more code comment to convince people that `HiveConf.getConfSystemProperties` contains only the --hiveconf?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587581230
**[Test build #118644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118644/testReport)** for PR 27041 at commit [`3718df9`](https://github.com/apache/spark/commit/3718df989c02641f864432b246313a158a11b9e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574393271
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573534273
Could you check the UT failure, @ajithme ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r365991246
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,9 +177,14 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to loose command line overwritten properties
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme removed a comment on issue #27041: [SPARK-25061]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme removed a comment on issue #27041: [SPARK-25061] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-570187234
gentle ping @HyukjinKwon @steveloughran @cloud-fan @yhuai @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574234640
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21490/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574227761
**[Test build #116713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116713/testReport)** for PR 27041 at commit [`5043b0c`](https://github.com/apache/spark/commit/5043b0caf2c63b3d37eb6fba7c4d0efbca697ab3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574234628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21327/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r365555097
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,9 +177,14 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to loose command line overwritten properties
Review comment:
According to the context, do you mean `lose` instead of `loose`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27041: [SPARK-25061] Precedence
for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27041: [SPARK-25061] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569510200
I think @yhuai wrote the comment above about why it's processed in that order, so may be able to review better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574224827
@dongjoon-hyun I have updated the PR with testcase failure also added a UT to reproduce and verify the issue. Please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574393271
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574323108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116713/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573883231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116656/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-588133485
I have no stong option about case 4.
BTW, you should also pay attention to `SparkSQLCLIDriver` where the hive `sesstionState` is initialized before the spark context
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-588113516
Thanks for pinging me @cloud-fan
We can override hive configurations in many ways.
Take `hive.metastore.uris` for an example, we can reset it via `--conf spark.hadoop.hive.metastore.uris=thrift://example.com:9083`, or `--conf spark.hive.hive.metastore.uris=thrift://example.com:9083` or `--hiveconf hive.metastore.uris=thrift://example.com:9083` or maybe(not sure) `--conf spark.driver.extraJavaOptions=-Dxxx`.
This PR seems to prefer `--hiveconf` than others.
Personally, I prefer those spark configurations always have higher precedence than other type
configurations including hive/hadoop/java/system, etc, as we are writing spark applicaitons.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573555164
> Could you check the UT failure, @ajithme ?
Sure, will update shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339959
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061]
Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569493029
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574323097
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574234640
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21490/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061] Precedence
for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-571909273
gentle ping @yhuai @dongjoon-hyun @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-588095894
@wangyum @yaooqinn can you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-588125267
> Thanks for pinging me @cloud-fan
>
> We can override hive configurations in many ways.
>
> Take `hive.metastore.uris` for an example, we can reset it via `--conf spark.hadoop.hive.metastore.uris=thrift://example.com:9083`, or `--conf spark.hive.hive.metastore.uris=thrift://example.com:9083` or `--hiveconf hive.metastore.uris=thrift://example.com:9083` or maybe(not sure) `--conf spark.driver.extraJavaOptions=-Dxxx`.
>
> This PR seems to prefer `--hiveconf` than others.
>
> Personally, I prefer those spark configurations always have higher precedence than other type
> configurations including hive/hadoop/java/system, etc, as we are writing spark applications.
Thanks @yaooqinn for your thoughts. This seems little confusing to know who is overriding as per the documentation mentioned in https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html#running-the-thrift-jdbcodbc-server
I agree with your opinion of having sparkConf as most precedence, but command line (--hiveconf) should be preferred over config file (hive-site.xml)
For the case you mentioned (X marks the conf is used)
type | case 1 | case 2 | case 3 | case 4
------------ | ------------- | ------------- | ------------- | -------------
--conf spark.hadoop.hive.* | X | X | - | -
--conf spark.hive.hive.* | X | - | X | -
--hiveconf | X | X | X | X
hive-site.xml | X | X | X | X
**Preference** | ? | --conf spark.hadoop.hive.* | --conf spark.hive.* | --hiveconf
so do you mean, in case 1, 2, 3 where spark.* conf is used it must get preference.?
how about case 4.?
i prefer in case 4 --hiveconf has precedence and rest cases spark conf can have higher precedence
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r366545134
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,8 +177,12 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
+ sparkConf.getAll.toMap ++ overriddenHiveProps ++ extraConfig).toMap
Review comment:
Sure, even without my changes in this PR, the `HiveConf` always considers the hive confs in system properties which were not set via `--hiveconf` as part of `HiveConf` constructor i.e Refer https://github.com/apache/hive/blob/rel/release-2.3.5/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4079 (same behaviour for hive 1.2.1 as well) hence, this do not change any flow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587581917
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574393279
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116725/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587238739
gentle ping @dongjoon-hyun @yhuai @cloud-fan @HyukjinKwon can we get this fix in 3.0.?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r380731555
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -176,9 +176,18 @@ private[hive] class HiveClientImpl(
// is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
- // 3: we set all entries in config to this hiveConf.
+ // 3: we take the conf passed as --hiveconf which would be set as system properties
+ // by org.apache.hive.service.server.HiveServer2.ServerOptionsProcessor.parse in
+ // org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main.
+ // 4: we set all entries in extraConfig to this hiveConf which have the highest precedence.
+ // To summarize, the order of precedence will be
+ // hadoopConf < sparkConf < overrideProps < extraConfig
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
Review comment:
@cloud-fan updated. is the comment adequate now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507255
**[Test build #118644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118644/testReport)** for PR 27041 at commit [`3718df9`](https://github.com/apache/spark/commit/3718df989c02641f864432b246313a158a11b9e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-576114698
gentle ping @dongjoon-hyun @yhuai
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339967
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21500/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574323108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116713/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573883221
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377898
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573385177
**[Test build #116541 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116541/testReport)** for PR 27041 at commit [`6969ec2`](https://github.com/apache/spark/commit/6969ec290fc6981aeb39a3eeb2f0d8d878ef4b9e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377825
**[Test build #116541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116541/testReport)** for PR 27041 at commit [`6969ec2`](https://github.com/apache/spark/commit/6969ec290fc6981aeb39a3eeb2f0d8d878ef4b9e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377568
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574322276
**[Test build #116713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116713/testReport)** for PR 27041 at commit [`5043b0c`](https://github.com/apache/spark/commit/5043b0caf2c63b3d37eb6fba7c4d0efbca697ab3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377900
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21327/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r366534585
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,8 +177,12 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
+ sparkConf.getAll.toMap ++ overriddenHiveProps ++ extraConfig).toMap
Review comment:
@yhuai Sure, I have updated the PR with reasonable pointers for the order. Does it suffice it now.?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573828076
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r380460761
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -176,9 +176,18 @@ private[hive] class HiveClientImpl(
// is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
- // 3: we set all entries in config to this hiveConf.
+ // 3: we take the conf passed as --hiveconf which would be set as system properties
+ // by org.apache.hive.service.server.HiveServer2.ServerOptionsProcessor.parse in
+ // org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main.
+ // 4: we set all entries in extraConfig to this hiveConf which have the highest precedence.
+ // To summarize, the order of precedence will be
+ // hadoopConf < sparkConf < overrideProps < extraConfig
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
Review comment:
so we totally ignore the --hive-conf previously?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507255
**[Test build #118644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118644/testReport)** for PR 27041 at commit [`3718df9`](https://github.com/apache/spark/commit/3718df989c02641f864432b246313a158a11b9e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507799
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061]
Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061] Spark SQL Thrift Server fails to not pick up hiveconf passing parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569493116
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587581937
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118644/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-576114698
gentle ping @dongjoon-hyun @yhuai
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573827546
**[Test build #116656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116656/testReport)** for PR 27041 at commit [`cd704bd`](https://github.com/apache/spark/commit/cd704bdd87b2c978ee0ebc76d3a2a9aeb0092453).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507799
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587507816
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23397/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574234628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573385210
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573828085
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21435/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573828076
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
ajithme commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r380461750
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -176,9 +176,18 @@ private[hive] class HiveClientImpl(
// is not set to builtin. When spark.sql.hive.metastore.jars is builtin, the classpath
// has hive-site.xml. So, HiveConf will use that to override its default values.
// 2: we set all spark confs to this hiveConf.
- // 3: we set all entries in config to this hiveConf.
+ // 3: we take the conf passed as --hiveconf which would be set as system properties
+ // by org.apache.hive.service.server.HiveServer2.ServerOptionsProcessor.parse in
+ // org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main.
+ // 4: we set all entries in extraConfig to this hiveConf which have the highest precedence.
+ // To summarize, the order of precedence will be
+ // hadoopConf < sparkConf < overrideProps < extraConfig
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
Review comment:
it was handled as --hiveconf as part of HiveConf constructor i.e Refer https://github.com/apache/hive/blob/rel/release-2.3.5/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4079 ,
i.e first it loads hive-site and then it adds --hiveconf properties on top.
But in spark we again add hadoopConf on top of it hence overwriting HiveConf order
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574323097
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574227761
**[Test build #116713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116713/testReport)** for PR 27041 at commit [`5043b0c`](https://github.com/apache/spark/commit/5043b0caf2c63b3d37eb6fba7c4d0efbca697ab3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573385211
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116541/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339967
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21500/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573828085
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21435/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-587581917
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574393279
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116725/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574392703
**[Test build #116725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116725/testReport)** for PR 27041 at commit [`2ee9f61`](https://github.com/apache/spark/commit/2ee9f617b88bef4cd6ba5b2b4c293837bd3dc6bb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041:
[SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-569493116
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573377898
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-573882628
**[Test build #116656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116656/testReport)** for PR 27041 at commit [`cd704bd`](https://github.com/apache/spark/commit/cd704bdd87b2c978ee0ebc76d3a2a9aeb0092453).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#issuecomment-574339959
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org