You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/01/12 09:15:17 UTC
[GitHub] [incubator-kyuubi] zhouyifan279 opened a new pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
zhouyifan279 opened a new pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739
### _Why are the changes needed?_
Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens to avoid errors appears in log repeatedly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#discussion_r783744072
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -80,39 +85,17 @@ class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with
object HadoopFsDelegationTokenProvider extends Logging {
- def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
- // Avoid unnecessary disk io by not loading default resources
- val newConf = KyuubiHadoopUtils.newHadoopConf(
- kyuubiConf,
- loadDefaults = false)
-
- hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
-
- hadoopFSsToAccess(kyuubiConf, hadoopConf)
- .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
- newConf
- }
-
- def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
- val filesystemsToAccess = kyuubiConf
- .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
- .flatMap { uri =>
- Try(new Path(uri).getFileSystem(hadoopConf)) match {
- case Success(value) =>
- Some(value)
- case Failure(e) =>
- warn(s"Failed to get Hadoop FileSystem instance by URI: $uri", e)
- None
- }
+ def validatedFsUris(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Seq[URI] = {
+ val uris = kyuubiConf.get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS).map(new URI(_)) :+
+ FileSystem.getDefaultUri(hadoopConf)
Review comment:
@ulysses-you Updated code to hand exception caused by URI format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#discussion_r783005277
##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -80,39 +85,17 @@ class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with
object HadoopFsDelegationTokenProvider extends Logging {
- def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
- // Avoid unnecessary disk io by not loading default resources
- val newConf = KyuubiHadoopUtils.newHadoopConf(
- kyuubiConf,
- loadDefaults = false)
-
- hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
-
- hadoopFSsToAccess(kyuubiConf, hadoopConf)
- .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
- newConf
- }
-
- def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
- val filesystemsToAccess = kyuubiConf
- .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
- .flatMap { uri =>
- Try(new Path(uri).getFileSystem(hadoopConf)) match {
- case Success(value) =>
- Some(value)
- case Failure(e) =>
- warn(s"Failed to get Hadoop FileSystem instance by URI: $uri", e)
- None
- }
+ def validatedFsUris(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Seq[URI] = {
+ val uris = kyuubiConf.get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS).map(new URI(_)) :+
+ FileSystem.getDefaultUri(hadoopConf)
Review comment:
what happens if `FileSystem.getDefaultUri` throw exception ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010966507
> can you provide some error logs in PR description?
Updated description with error logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 edited a comment on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
zhouyifan279 edited a comment on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010966507
> can you provide some error logs in PR description?
@ulysses-you Updated description with error logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1011918441
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1739](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6d80b8e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/a550ac705bc92c60322524acf1b36c4f981e3a6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a550ac7) will **decrease** coverage by `0.02%`.
> The diff coverage is `94.11%`.
> :exclamation: Current head 6d80b8e differs from pull request most recent head 387036d. Consider uploading reports for the commit 387036d to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1739 +/- ##
============================================
- Coverage 59.55% 59.53% -0.03%
+ Complexity 234 233 -1
============================================
Files 268 268
Lines 13314 13307 -7
Branches 1692 1690 -2
============================================
- Hits 7929 7922 -7
Misses 4708 4708
Partials 677 677
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `94.73% <94.11%> (+1.40%)` | :arrow_up: |
| [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `92.50% <0.00%> (-2.50%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a550ac7...387036d](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012090373
thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012091198
there is some conflicts with branch-1.4 @zhouyifan279 can you send a pr for branch-1.4 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010820983
cc @ulysses-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012059462
cc @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010892179
can you provide some error logs in PR description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org