You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/01/12 09:15:17 UTC

[GitHub] [incubator-kyuubi] zhouyifan279 opened a new pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

zhouyifan279 opened a new pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739


   ### _Why are the changes needed?_
   Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens to avoid errors appears in log repeatedly.
   
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhouyifan279 commented on a change in pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#discussion_r783744072



##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -80,39 +85,17 @@ class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with
 
 object HadoopFsDelegationTokenProvider extends Logging {
 
-  def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
-    // Avoid unnecessary disk io by not loading default resources
-    val newConf = KyuubiHadoopUtils.newHadoopConf(
-      kyuubiConf,
-      loadDefaults = false)
-
-    hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
-
-    hadoopFSsToAccess(kyuubiConf, hadoopConf)
-      .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
-    newConf
-  }
-
-  def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
-    val filesystemsToAccess = kyuubiConf
-      .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
-      .flatMap { uri =>
-        Try(new Path(uri).getFileSystem(hadoopConf)) match {
-          case Success(value) =>
-            Some(value)
-          case Failure(e) =>
-            warn(s"Failed to get Hadoop FileSystem instance by URI: $uri", e)
-            None
-        }
+  def validatedFsUris(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Seq[URI] = {
+    val uris = kyuubiConf.get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS).map(new URI(_)) :+
+      FileSystem.getDefaultUri(hadoopConf)

Review comment:
       @ulysses-you Updated code to hand exception caused by URI format




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#discussion_r783005277



##########
File path: kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HadoopFsDelegationTokenProvider.scala
##########
@@ -80,39 +85,17 @@ class HadoopFsDelegationTokenProvider extends HadoopDelegationTokenProvider with
 
 object HadoopFsDelegationTokenProvider extends Logging {
 
-  def disableFsCache(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Configuration = {
-    // Avoid unnecessary disk io by not loading default resources
-    val newConf = KyuubiHadoopUtils.newHadoopConf(
-      kyuubiConf,
-      loadDefaults = false)
-
-    hadoopConf.iterator().asScala.foreach(e => newConf.set(e.getKey, e.getValue))
-
-    hadoopFSsToAccess(kyuubiConf, hadoopConf)
-      .foreach(fs => newConf.setBoolean(s"fs.${fs.getScheme}.impl.disable.cache", true))
-    newConf
-  }
-
-  def hadoopFSsToAccess(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Set[FileSystem] = {
-    val filesystemsToAccess = kyuubiConf
-      .get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS)
-      .flatMap { uri =>
-        Try(new Path(uri).getFileSystem(hadoopConf)) match {
-          case Success(value) =>
-            Some(value)
-          case Failure(e) =>
-            warn(s"Failed to get Hadoop FileSystem instance by URI: $uri", e)
-            None
-        }
+  def validatedFsUris(kyuubiConf: KyuubiConf, hadoopConf: Configuration): Seq[URI] = {
+    val uris = kyuubiConf.get(KyuubiConf.CREDENTIALS_HADOOP_FS_URIS).map(new URI(_)) :+
+      FileSystem.getDefaultUri(hadoopConf)

Review comment:
       what happens if `FileSystem.getDefaultUri` throw exception ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010966507


   > can you provide some error logs in PR description?
   
   Updated description with error logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhouyifan279 edited a comment on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
zhouyifan279 edited a comment on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010966507


   > can you provide some error logs in PR description?
   
   @ulysses-you Updated description with error logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1011918441


   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#1739](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6d80b8e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/a550ac705bc92c60322524acf1b36c4f981e3a6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a550ac7) will **decrease** coverage by `0.02%`.
   > The diff coverage is `94.11%`.
   
   > :exclamation: Current head 6d80b8e differs from pull request most recent head 387036d. Consider uploading reports for the commit 387036d to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739/graphs/tree.svg?width=650&height=150&src=pr&token=925D4tb9AH&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1739      +/-   ##
   ============================================
   - Coverage     59.55%   59.53%   -0.03%     
   + Complexity      234      233       -1     
   ============================================
     Files           268      268              
     Lines         13314    13307       -7     
     Branches       1692     1690       -2     
   ============================================
   - Hits           7929     7922       -7     
     Misses         4708     4708              
     Partials        677      677              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../credentials/HadoopFsDelegationTokenProvider.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IYWRvb3BGc0RlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | `94.73% <94.11%> (+1.40%)` | :arrow_up: |
   | [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `92.50% <0.00%> (-2.50%)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a550ac7...387036d](https://codecov.io/gh/apache/incubator-kyuubi/pull/1739?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012090373


   thanks, merging to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you closed pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012091198


   there is some conflicts with branch-1.4 @zhouyifan279 can you send a pr for branch-1.4 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010820983


   cc @ulysses-you 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhouyifan279 commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1012059462


   cc @pan3793 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1739: [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1739:
URL: https://github.com/apache/incubator-kyuubi/pull/1739#issuecomment-1010892179


   can you provide some error logs in PR description?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org