You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2023/05/13 04:50:35 UTC
[GitHub] [kyuubi] turboFei opened a new pull request, #4835: Using hive conf to check whether HIVE_DELEGATION_TOKEN is needed
turboFei opened a new pull request, #4835:
URL: https://github.com/apache/kyuubi/pull/4835
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4835:
URL: https://github.com/apache/kyuubi/pull/4835#issuecomment-1546539286
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/4835?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#4835](https://app.codecov.io/gh/apache/kyuubi/pull/4835?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7657cbb) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/1e310a0818f23dd7b8f2d4afd17795794fed46ea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (1e310a0) will **increase** coverage by `0.03%`.
> The diff coverage is `80.00%`.
```diff
@@ Coverage Diff @@
## master #4835 +/- ##
============================================
+ Coverage 58.03% 58.07% +0.03%
Complexity 13 13
============================================
Files 583 583
Lines 32561 32570 +9
Branches 4318 4320 +2
============================================
+ Hits 18897 18914 +17
+ Misses 11845 11836 -9
- Partials 1819 1820 +1
```
| [Impacted Files](https://app.codecov.io/gh/apache/kyuubi/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ubi/engine/spark/SparkTBinaryFrontendService.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9TcGFya1RCaW5hcnlGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `81.63% <80.00%> (-0.39%)` | :arrow_down: |
... and [6 files with indirect coverage changes](https://app.codecov.io/gh/apache/kyuubi/pull/4835/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #4835:
URL: https://github.com/apache/kyuubi/pull/4835#issuecomment-1547118875
Not sure why hive-site.xml is not loaded.
<img width="1580" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/99de73d0-3519-4af3-8f0a-90967949ec0e">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #4835:
URL: https://github.com/apache/kyuubi/pull/4835#issuecomment-1547133512
@pan3793 updated the pr description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #4835:
URL: https://github.com/apache/kyuubi/pull/4835#issuecomment-1547130068
```
/**
* Return an appropriate (subclass) of Configuration. Creating config can initialize some Hadoop
* subsystems.
*/
def newConfiguration(conf: SparkConf): Configuration = {
val hadoopConf = SparkHadoopUtil.newConfiguration(conf)
hadoopConf.addResource(SparkHadoopUtil.SPARK_HADOOP_CONF_FILE)
hadoopConf
}
```
```
/**
* Name of the file containing the gateway's Hadoop configuration, to be overlayed on top of the
* cluster's Hadoop config. It is up to the Spark code launching the application to create
* this file if it's desired. If the file doesn't exist, it will just be ignored.
*/
private[spark] val SPARK_HADOOP_CONF_FILE = "__spark_hadoop_conf__.xml"
```
Seems for spark on k8s, there is no file named `__spark_hadoop_conf__.xml`
```
/**
* Return an appropriate (subclass) of Configuration. Creating config can initialize some Hadoop
* subsystems.
*/
def newConfiguration(conf: SparkConf): Configuration = {
val hadoopConf = SparkHadoopUtil.newConfiguration(conf)
hadoopConf.addResource(SparkHadoopUtil.SPARK_HADOOP_CONF_FILE)
hadoopConf
}
```
```
/**
* Name of the file containing the gateway's Hadoop configuration, to be overlayed on top of the
* cluster's Hadoop config. It is up to the Spark code launching the application to create
* this file if it's desired. If the file doesn't exist, it will just be ignored.
*/
private[spark] val SPARK_HADOOP_CONF_FILE = "__spark_hadoop_conf__.xml"
```
Seems for spark on k8s, there is no file named `__spark_hadoop_conf__.xml`
![Uploading image.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4835:
URL: https://github.com/apache/kyuubi/pull/4835#issuecomment-1546931633
> For spark on k8s, the hive-site.xml is placed to working directory after spark context launched. See details in https://github.com/apache/spark/pull/37417
Sorry, I don't get your point, w/ https://github.com/apache/spark/pull/37417, Spark downloads and adds those files during spark submit phase, which should happen before SparkContext initialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei closed pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #4835: [K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
URL: https://github.com/apache/kyuubi/pull/4835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org