You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/09/06 01:37:47 UTC
[GitHub] [incubator-kyuubi] zhouyifan279 opened a new pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
zhouyifan279 opened a new pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029
### _Why are the changes needed?_
This PR finishes the work of task #1008
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1029: [WIP][KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913377278
@pan3793 @yaooqinn Ready to be reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702799757
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ private var internalConf: Configuration = _
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ UserGroupInformation.isSecurityEnabled
Review comment:
shall we call `UserGroupInformation.setConfiguration` before ?
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ private var internalConf: Configuration = _
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // A FileSystem object are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are cached at every method invocation.
+ if (internalConf == null) {
+ internalConf = disableFsCache(kyuubiConf, hadoopConf)
Review comment:
why we cache `internalConf ` but not cache `filesystem`. isn't it should be same ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1029: [WIP][KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702539411
##########
File path: pom.xml
##########
@@ -985,6 +985,12 @@
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
Review comment:
We don't need it, it's already covered by `hadoop-client-api` and `hadoop-client-runtime`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702837072
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ private var internalConf: Configuration = _
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ UserGroupInformation.isSecurityEnabled
Review comment:
Change `UserGroupInformation#conf` might be a little risky. How about `SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-914114484
@ulysses-you modified according to advice. Ready to be reviewed again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [WIP][KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c11ba6e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/057bac19136466da5adb5149506a09b451672d30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (057bac1) will **increase** coverage by `0.11%`.
> The diff coverage is `97.50%`.
> :exclamation: Current head c11ba6e differs from pull request most recent head 84e12f9. Consider uploading reports for the commit 84e12f9 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
+ Coverage 80.65% 80.77% +0.11%
Complexity 14 14
============================================
Files 159 160 +1
Lines 5837 5877 +40
Branches 657 657
============================================
+ Hits 4708 4747 +39
Misses 734 734
- Partials 395 396 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.42% <96.42%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [057bac1...84e12f9](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8d1451d) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **decrease** coverage by `0.00%`.
> The diff coverage is `89.65%`.
> :exclamation: Current head 8d1451d differs from pull request most recent head da50aae. Consider uploading reports for the commit da50aae to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 78.56% 78.56% -0.01%
Complexity 61 61
============================================
Files 167 168 +1
Lines 6196 6241 +45
Branches 723 724 +1
============================================
+ Hits 4868 4903 +35
- Misses 903 914 +11
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/kyuubi/ha/client/ServiceDiscovery.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL2NsaWVudC9TZXJ2aWNlRGlzY292ZXJ5LnNjYWxh) | `45.52% <33.33%> (ø)` | |
| [...g/apache/kyuubi/session/KyuubiSessionManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25NYW5hZ2VyLnNjYWxh) | `97.05% <66.66%> (ø)` | |
| [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `78.94% <77.77%> (-0.28%)` | :arrow_down: |
| [...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL0hpZ2hBdmFpbGFiaWxpdHlDb25mLnNjYWxh) | `98.27% <92.30%> (ø)` | |
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...a/org/apache/spark/kyuubi/SparkContextHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsva3l1dWJpL1NwYXJrQ29udGV4dEhlbHBlci5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../apache/kyuubi/client/KyuubiSyncThriftClient.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jbGllbnQvS3l1dWJpU3luY1RocmlmdENsaWVudC5zY2FsYQ==) | `82.02% <100.00%> (+0.20%)` | :arrow_up: |
| [.../org/apache/kyuubi/events/KyuubiSessionEvent.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9ldmVudHMvS3l1dWJpU2Vzc2lvbkV2ZW50LnNjYWxh) | `92.00% <100.00%> (-0.31%)` | :arrow_down: |
| [.../org/apache/kyuubi/session/KyuubiSessionImpl.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25JbXBsLnNjYWxh) | `89.58% <100.00%> (+1.77%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...da50aae](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ed9739a) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **decrease** coverage by `0.04%`.
> The diff coverage is `97.43%`.
> :exclamation: Current head ed9739a differs from pull request most recent head bf7c628. Consider uploading reports for the commit bf7c628 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 78.56% 78.52% -0.05%
Complexity 61 61
============================================
Files 167 168 +1
Lines 6196 6235 +39
Branches 723 723
============================================
+ Hits 4868 4896 +28
- Misses 903 915 +12
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `38.18% <0.00%> (-16.37%)` | :arrow_down: |
| [.../scala/org/apache/kyuubi/server/KyuubiServer.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpU2VydmVyLnNjYWxh) | `56.52% <0.00%> (-2.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...bf7c628](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [WIP][KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702601598
##########
File path: pom.xml
##########
@@ -985,6 +985,12 @@
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
Review comment:
Checked that hadoop-hdfs is included in org.apache.hadoop:hadoop-client-minicluster.
Removed hadoop-hdfs dependency here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703147446
##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
##########
@@ -249,6 +249,22 @@ object KyuubiConf {
.checkValue(t => t > 0, "must be positive integer")
.createWithDefault(Duration.ofMinutes(1).toMillis)
+ val CREDENTIALS_HADOOP_FS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("credentials.hadoopfs.enabled")
+ .doc("Whether to renew HadoopFS DelegationToken")
+ .version("1.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val CREDENTIALS_HADOOP_FS_URLS: ConfigEntry[Seq[String]] =
+ buildConf("credentials.hadoopfs.urls")
Review comment:
In reference to the [discussion](https://github.com/apache/spark/pull/16432#issuecomment-270320823) about the serviceName of `HadoopFSCredentialProvider` in Spark, using `hadoopfs` to stand for Hadoop Compatible file systems seems to be more appropriate.
Maybe we, should rename `credentials.hadoopfs.urls` to `credentials.hadoopfs.uris` as they are parsed to java.net.URI objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703133392
##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
##########
@@ -249,6 +249,22 @@ object KyuubiConf {
.checkValue(t => t > 0, "must be positive integer")
.createWithDefault(Duration.ofMinutes(1).toMillis)
+ val CREDENTIALS_HADOOP_FS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("credentials.hadoopfs.enabled")
+ .doc("Whether to renew HadoopFS DelegationToken")
+ .version("1.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val CREDENTIALS_HADOOP_FS_URLS: ConfigEntry[Seq[String]] =
+ buildConf("credentials.hadoopfs.urls")
Review comment:
How about
```
credentials.hadoop.filesystems
credentials.hadoop.enabled
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-914133459
thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61c1220) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/057bac19136466da5adb5149506a09b451672d30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (057bac1) will **decrease** coverage by `2.11%`.
> The diff coverage is `47.48%`.
> :exclamation: Current head 61c1220 differs from pull request most recent head 107ce12. Consider uploading reports for the commit 107ce12 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 80.65% 78.54% -2.12%
- Complexity 14 61 +47
============================================
Files 159 168 +9
Lines 5837 6235 +398
Branches 657 723 +66
============================================
+ Hits 4708 4897 +189
- Misses 734 914 +180
- Partials 395 424 +29
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ache/kyuubi/sql/zorder/KyuubiZorderExtension.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvS3l1dWJpWm9yZGVyRXh0ZW5zaW9uLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/sql/zorder/Zorder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyLnNjYWxh) | `15.00% <15.00%> (ø)` | |
| [...apache/kyuubi/sql/zorder/ZorderSqlAstBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyU3FsQXN0QnVpbGRlci5zY2FsYQ==) | `31.57% <31.57%> (ø)` | |
| [...rg/apache/kyuubi/sql/zorder/ZorderBytesUtils.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyQnl0ZXNVdGlscy5zY2FsYQ==) | `41.66% <41.66%> (ø)` | |
| [...rg/apache/kyuubi/sql/KyuubiSparkSQLExtension.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlTcGFya1NRTEV4dGVuc2lvbi5zY2FsYQ==) | `88.88% <50.00%> (-11.12%)` | :arrow_down: |
| [...bi/sql/zorder/ZorderSparkSqlExtensionsParser.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyU3BhcmtTcWxFeHRlbnNpb25zUGFyc2VyLnNjYWxh) | `68.08% <68.08%> (ø)` | |
| [...g/apache/kyuubi/sql/zorder/ZorderBeforeWrite.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyQmVmb3JlV3JpdGUuc2NhbGE=) | `74.07% <74.07%> (ø)` | |
| [...ache/kyuubi/sql/zorder/OptimizeZorderCommand.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvT3B0aW1pemVab3JkZXJDb21tYW5kLnNjYWxh) | `81.81% <81.81%> (ø)` | |
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...org/apache/kyuubi/sql/zorder/ZorderException.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyRXhjZXB0aW9uLnNjYWxh) | `100.00% <100.00%> (ø)` | |
| ... and [10 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [057bac1...107ce12](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8d1451d) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **decrease** coverage by `0.00%`.
> The diff coverage is `89.65%`.
> :exclamation: Current head 8d1451d differs from pull request most recent head 695cd95. Consider uploading reports for the commit 695cd95 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 78.56% 78.56% -0.01%
Complexity 61 61
============================================
Files 167 168 +1
Lines 6196 6241 +45
Branches 723 724 +1
============================================
+ Hits 4868 4903 +35
- Misses 903 914 +11
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/kyuubi/ha/client/ServiceDiscovery.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL2NsaWVudC9TZXJ2aWNlRGlzY292ZXJ5LnNjYWxh) | `45.52% <33.33%> (ø)` | |
| [...g/apache/kyuubi/session/KyuubiSessionManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25NYW5hZ2VyLnNjYWxh) | `97.05% <66.66%> (ø)` | |
| [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `78.94% <77.77%> (-0.28%)` | :arrow_down: |
| [...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL0hpZ2hBdmFpbGFiaWxpdHlDb25mLnNjYWxh) | `98.27% <92.30%> (ø)` | |
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...a/org/apache/spark/kyuubi/SparkContextHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsva3l1dWJpL1NwYXJrQ29udGV4dEhlbHBlci5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../apache/kyuubi/client/KyuubiSyncThriftClient.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jbGllbnQvS3l1dWJpU3luY1RocmlmdENsaWVudC5zY2FsYQ==) | `82.02% <100.00%> (+0.20%)` | :arrow_up: |
| [.../org/apache/kyuubi/events/KyuubiSessionEvent.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9ldmVudHMvS3l1dWJpU2Vzc2lvbkV2ZW50LnNjYWxh) | `92.00% <100.00%> (-0.31%)` | :arrow_down: |
| [.../org/apache/kyuubi/session/KyuubiSessionImpl.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25JbXBsLnNjYWxh) | `89.58% <100.00%> (+1.77%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...695cd95](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703130896
##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
##########
@@ -249,6 +249,22 @@ object KyuubiConf {
.checkValue(t => t > 0, "must be positive integer")
.createWithDefault(Duration.ofMinutes(1).toMillis)
+ val CREDENTIALS_HADOOP_FS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("credentials.hadoopfs.enabled")
+ .doc("Whether to renew HadoopFS DelegationToken")
+ .version("1.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val CREDENTIALS_HADOOP_FS_URLS: ConfigEntry[Seq[String]] = KyuubiConf
Review comment:
rm KyuubiConf
##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
##########
@@ -249,6 +249,22 @@ object KyuubiConf {
.checkValue(t => t > 0, "must be positive integer")
.createWithDefault(Duration.ofMinutes(1).toMillis)
+ val CREDENTIALS_HADOOP_FS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("credentials.hadoopfs.enabled")
+ .doc("Whether to renew HadoopFS DelegationToken")
+ .version("1.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val CREDENTIALS_HADOOP_FS_URLS: ConfigEntry[Seq[String]] = KyuubiConf
+ .buildConf("credentials.hadoopfs.urls")
+ .doc("Extra Hadoop filesystem URLs for which to request delegation tokens. " +
Review comment:
indent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703194137
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
Review comment:
why not use kyuubi server user `UserGroupInformation.getCurrentUser().getUserName()` as the renewer ?
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
+ }
+ } finally {
+ // Token renewal interval is longer than FileSystems' underlying connections' max idle time.
+ // Close FileSystems won't lose efficiency.
+ fileSystems.foreach(_.close())
+ }
+ }
+ }
+
+}
+
+object HadoopFsDelegationTokenProvider {
+
+ def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
+ // Avoid unnecessary disk io by not loading default resources
+ val newConf = new Configuration(false)
+ hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
+
+ hadoopFSsToAccess(kyuubiConf, hadoopConf)
+ .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
+ newConf
+ }
+
+ def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
+ val defaultFS = FileSystem.get(hadoopConf)
+ val filesystemsToAccess = kyuubiConf
+ .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
+ .map(new Path(_).getFileSystem(hadoopConf))
+ .toSet
+
+ filesystemsToAccess + defaultFS
+ }
+
+ def doAsProxyUser[T](proxyUser: String)(f: => T): T = {
+ // For some reason the Scala-generated anonymous class ends up causing an
+ // UndeclaredThrowableException, even if you annotate the method with @throws.
+ try {
+ val ugi = UserGroupInformation.createProxyUser(proxyUser, UserGroupInformation.getCurrentUser)
+ ugi.doAs(new PrivilegedExceptionAction[T] {
+ override def run(): T = f
+ })
+ } catch {
+ case e: UndeclaredThrowableException => throw Option(e.getCause).getOrElse(e)
Review comment:
what happen if we fail to get token for one user, does it affects other user ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702619252
##########
File path: kyuubi-server/pom.xml
##########
@@ -164,6 +164,12 @@
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </exclusion>
+ </exclusions>
Review comment:
Waiting #1028, would be ready today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703298933
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
Review comment:
We actually obtain a new token every time. Renewer is not needed.
Anyway, setting a renewer can avoid potential NPE. I'll set renewer to `UserGroupInformation.getCurrentUser().getUserName()`.
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
Review comment:
We actually obtain a new token every time. Renewer is not needed.
Anyway, setting a renewer can avoid potential NPE.
I'll set renewer to `UserGroupInformation.getCurrentUser().getUserName()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703303055
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
+ }
+ } finally {
+ // Token renewal interval is longer than FileSystems' underlying connections' max idle time.
+ // Close FileSystems won't lose efficiency.
+ fileSystems.foreach(_.close())
+ }
+ }
+ }
+
+}
+
+object HadoopFsDelegationTokenProvider {
+
+ def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
+ // Avoid unnecessary disk io by not loading default resources
+ val newConf = new Configuration(false)
+ hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
+
+ hadoopFSsToAccess(kyuubiConf, hadoopConf)
+ .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
+ newConf
+ }
+
+ def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
+ val defaultFS = FileSystem.get(hadoopConf)
+ val filesystemsToAccess = kyuubiConf
+ .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
+ .map(new Path(_).getFileSystem(hadoopConf))
+ .toSet
+
+ filesystemsToAccess + defaultFS
+ }
+
+ def doAsProxyUser[T](proxyUser: String)(f: => T): T = {
+ // For some reason the Scala-generated anonymous class ends up causing an
+ // UndeclaredThrowableException, even if you annotate the method with @throws.
+ try {
+ val ugi = UserGroupInformation.createProxyUser(proxyUser, UserGroupInformation.getCurrentUser)
+ ugi.doAs(new PrivilegedExceptionAction[T] {
+ override def run(): T = f
+ })
+ } catch {
+ case e: UndeclaredThrowableException => throw Option(e.getCause).getOrElse(e)
Review comment:
`HadoopCredentialsManager` will catch the exception and schedule a retry task.
Failures are controlled in user scope and will not affect other users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ed9739a) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **decrease** coverage by `0.04%`.
> The diff coverage is `97.43%`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 78.56% 78.52% -0.05%
Complexity 61 61
============================================
Files 167 168 +1
Lines 6196 6235 +39
Branches 723 723
============================================
+ Hits 4868 4896 +28
- Misses 903 915 +12
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `38.18% <0.00%> (-16.37%)` | :arrow_down: |
| [.../scala/org/apache/kyuubi/server/KyuubiServer.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpU2VydmVyLnNjYWxh) | `56.52% <0.00%> (-2.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...ed9739a](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703298933
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, SecurityUtil, UserGroupInformation}
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ SecurityUtil.getAuthenticationMethod(hadoopConf) != AuthenticationMethod.SIMPLE
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // FileSystem objects are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are created and cached at every method
+ // invocation.
+ val internalConf = disableFsCache(kyuubiConf, hadoopConf)
+
+ doAsProxyUser(owner) {
+ val fileSystems =
+ HadoopFsDelegationTokenProvider.hadoopFSsToAccess(kyuubiConf, internalConf)
+
+ try {
+ fileSystems.foreach { fs =>
+ info(s"getting token owned by $owner for: $fs")
+ fs.addDelegationTokens(null, creds)
Review comment:
We actually obtain a new token every time. Renewer is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (09a661c) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/057bac19136466da5adb5149506a09b451672d30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (057bac1) will **increase** coverage by `0.11%`.
> The diff coverage is `97.50%`.
> :exclamation: Current head 09a661c differs from pull request most recent head 107ce12. Consider uploading reports for the commit 107ce12 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
+ Coverage 80.65% 80.77% +0.11%
Complexity 14 14
============================================
Files 159 160 +1
Lines 5837 5877 +40
Branches 657 657
============================================
+ Hits 4708 4747 +39
Misses 734 734
- Partials 395 396 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.42% <96.42%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [057bac1...107ce12](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (09a661c) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/057bac19136466da5adb5149506a09b451672d30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (057bac1) will **increase** coverage by `0.11%`.
> The diff coverage is `97.50%`.
> :exclamation: Current head 09a661c differs from pull request most recent head 84e12f9. Consider uploading reports for the commit 84e12f9 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
+ Coverage 80.65% 80.77% +0.11%
Complexity 14 14
============================================
Files 159 160 +1
Lines 5837 5877 +40
Branches 657 657
============================================
+ Hits 4708 4747 +39
Misses 734 734
- Partials 395 396 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.42% <96.42%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [057bac1...84e12f9](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (695cd95) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **increase** coverage by `0.13%`.
> The diff coverage is `97.50%`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
+ Coverage 78.56% 78.70% +0.13%
Complexity 61 61
============================================
Files 167 169 +2
Lines 6196 6249 +53
Branches 723 725 +2
============================================
+ Hits 4868 4918 +50
- Misses 903 907 +4
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.42% <96.42%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `38.18% <0.00%> (-16.37%)` | :arrow_down: |
| [...bi/sql/zorder/ZorderSparkSqlExtensionsParser.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC96b3JkZXIvWm9yZGVyU3BhcmtTcWxFeHRlbnNpb25zUGFyc2VyLnNjYWxh) | `66.66% <0.00%> (-1.42%)` | :arrow_down: |
| [...org/apache/kyuubi/server/RestFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvUmVzdEZyb250ZW5kU2VydmljZS5zY2FsYQ==) | `83.92% <0.00%> (-0.69%)` | :arrow_down: |
| [.../org/apache/kyuubi/events/KyuubiSessionEvent.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9ldmVudHMvS3l1dWJpU2Vzc2lvbkV2ZW50LnNjYWxh) | `92.00% <0.00%> (-0.31%)` | :arrow_down: |
| [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `78.94% <0.00%> (-0.28%)` | :arrow_down: |
| [...la/org/apache/kyuubi/session/AbstractSession.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0Fic3RyYWN0U2Vzc2lvbi5zY2FsYQ==) | `95.40% <0.00%> (-0.11%)` | :arrow_down: |
| [...g/apache/kyuubi/session/KyuubiSessionManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25NYW5hZ2VyLnNjYWxh) | `97.05% <0.00%> (ø)` | |
| [.../apache/kyuubi/server/api/v1/ApiRootResource.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvYXBpL3YxL0FwaVJvb3RSZXNvdXJjZS5zY2FsYQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...695cd95](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61c1220) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f584953f7fdd60c3cb7589406859012891d847cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f584953) will **decrease** coverage by `0.02%`.
> The diff coverage is `97.43%`.
> :exclamation: Current head 61c1220 differs from pull request most recent head ed9739a. Consider uploading reports for the commit ed9739a to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
- Coverage 78.56% 78.54% -0.03%
Complexity 61 61
============================================
Files 167 168 +1
Lines 6196 6235 +39
Branches 723 723
============================================
+ Hits 4868 4897 +29
- Misses 903 914 +11
+ Partials 425 424 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.29% <96.29%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `38.18% <0.00%> (-16.37%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f584953...ed9739a](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913787541
#1028 got merged, it excluded all hadoop deps from hudi, pls revert the related changes and rebase on latest master branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r703147446
##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
##########
@@ -249,6 +249,22 @@ object KyuubiConf {
.checkValue(t => t > 0, "must be positive integer")
.createWithDefault(Duration.ofMinutes(1).toMillis)
+ val CREDENTIALS_HADOOP_FS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("credentials.hadoopfs.enabled")
+ .doc("Whether to renew HadoopFS DelegationToken")
+ .version("1.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val CREDENTIALS_HADOOP_FS_URLS: ConfigEntry[Seq[String]] =
+ buildConf("credentials.hadoopfs.urls")
Review comment:
In reference to the [discussion](https://github.com/apache/spark/pull/16432#issuecomment-270320823) about the serviceName of `HadoopFSCredentialProvider` in Spark, using `hadoopfs` to stand for Hadoop Compatible file systems seems to be more appropriate.
Maybe we, should rename `credentials.hadoopfs.urls` to `credentials.hadoopfs.uris` as they are parsed as java.net.URI objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702842639
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ private var internalConf: Configuration = _
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ UserGroupInformation.isSecurityEnabled
Review comment:
sgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#discussion_r702854477
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import java.lang.reflect.UndeclaredThrowableException
+import java.security.PrivilegedExceptionAction
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.{disableFsCache, doAsProxyUser}
+
+class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with Logging {
+
+ override val serviceName: String = "hadoopfs"
+
+ private var internalConf: Configuration = _
+
+ override def delegationTokensRequired(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf): Boolean = {
+ UserGroupInformation.isSecurityEnabled
+ }
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ kyuubiConf: KyuubiConf,
+ owner: String,
+ creds: Credentials): Unit = {
+ // A FileSystem object are cached in FileSystem.CACHE by a composite key.
+ // The UserGroupInformation object used to create it is part of that key.
+ // If cache is enabled, new FileSystem objects are cached at every method invocation.
+ if (internalConf == null) {
+ internalConf = disableFsCache(kyuubiConf, hadoopConf)
Review comment:
Number of cached FileSystem objects grows as kyuubi user number grows. To reduce memory usage under large number of users, FileSystem objects are closed after obtaining tokens.
Besides, although cached FileSystem objects maitains connections to NameNodes, connections are closed in 10 seconds by default. Token renewal won't benefit from cached FileSystem objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1029: [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913935458
@pan3793 Thanks for the notification. Rebased on the master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #1029: [WIP][KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1029:
URL: https://github.com/apache/incubator-kyuubi/pull/1029#issuecomment-913371104
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1029](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c11ba6e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/057bac19136466da5adb5149506a09b451672d30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (057bac1) will **increase** coverage by `0.11%`.
> The diff coverage is `97.50%`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1029 +/- ##
============================================
+ Coverage 80.65% 80.77% +0.11%
Complexity 14 14
============================================
Files 159 160 +1
Lines 5837 5877 +40
Branches 657 657
============================================
+ Hits 4708 4747 +39
Misses 734 734
- Partials 395 396 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `96.42% <96.42%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `95.25% <100.00%> (+0.11%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [057bac1...c11ba6e](https://codecov.io/gh/apache/incubator-kyuubi/pull/1029?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org