You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/07/30 00:13:11 UTC

[GitHub] [gobblin] ZihanLi58 opened a new pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

ZihanLi58 opened a new pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345


   
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1500
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   Previously we always check with one default yarn client, but if the cluster is enabled with Robin and can connect to different RM, it will ends up we cannot find existing application and submit duplicate tasks. So now we need to check against different potential yarn cluster, and also make sure when submitting yarn application, we only have one RM token.
   
   Options
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Test start a new application can work, test reconnect can work 
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a230fb0) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **increase** coverage by `2.43%`.
   > The diff coverage is `32.50%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   + Coverage     46.54%   48.97%   +2.43%     
   + Complexity    10136     8644    -1492     
   ============================================
     Files          2051     1671     -380     
     Lines         79547    64088   -15459     
     Branches       8880     7355    -1525     
   ============================================
   - Hits          37027    31389    -5638     
   + Misses        39091    29719    -9372     
   + Partials       3429     2980     -449     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `25.40% <32.50%> (+1.09%)` | :arrow_up: |
   | [...blin/data/management/copy/RecursivePathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvUmVjdXJzaXZlUGF0aEZpbmRlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...data/management/copy/extractor/EmptyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0VtcHR5RXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...che/gobblin/runtime/TaskStateCollectorService.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlQ29sbGVjdG9yU2VydmljZS5qYXZh) | `63.30% <0.00%> (-15.77%)` | :arrow_down: |
   | [...a/management/copy/hive/HiveLocationDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlTG9jYXRpb25EZXNjcmlwdG9yLmphdmE=) | `36.17% <0.00%> (-14.90%)` | :arrow_down: |
   | [...anagement/copy/hive/UnpartitionedTableFileSet.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9VbnBhcnRpdGlvbmVkVGFibGVGaWxlU2V0LmphdmE=) | `44.44% <0.00%> (-12.97%)` | :arrow_down: |
   | [...ache/gobblin/util/commit/DeleteFileCommitStep.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9jb21taXQvRGVsZXRlRmlsZUNvbW1pdFN0ZXAuamF2YQ==) | `34.04% <0.00%> (-8.52%)` | :arrow_down: |
   | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `52.67% <0.00%> (-8.46%)` | :arrow_down: |
   | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `67.54% <0.00%> (-6.63%)` | :arrow_down: |
   | ... and [410 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...a230fb0](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a230fb0) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `0.14%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   46.40%   -0.15%     
   + Complexity    10136     3069    -7067     
   ============================================
     Files          2051      644    -1407     
     Lines         79547    25091   -54456     
     Branches       8880     2996    -5884     
   ============================================
   - Hits          37027    11644   -25383     
   + Misses        39091    12195   -26896     
   + Partials       3429     1252    -2177     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...he/gobblin/service/modules/flow/FlowGraphPath.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L0Zsb3dHcmFwaFBhdGguamF2YQ==) | `81.15% <0.00%> (-2.67%)` | :arrow_down: |
   | [...in/service/modules/core/GobblinServiceManager.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dvYmJsaW5TZXJ2aWNlTWFuYWdlci5qYXZh) | `50.20% <0.00%> (-0.84%)` | :arrow_down: |
   | [...in/service/modules/orchestration/Orchestrator.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL09yY2hlc3RyYXRvci5qYXZh) | `53.10% <0.00%> (-0.35%)` | :arrow_down: |
   | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `58.59% <0.00%> (-0.27%)` | :arrow_down: |
   | [.../org/apache/gobblin/service/ServiceConfigKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9TZXJ2aWNlQ29uZmlnS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ava/org/apache/gobblin/util/hadoop/TokenUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvaGFkb29wL1Rva2VuVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.37% <0.00%> (ø)` | |
   | [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [.../destination/DestinationDatasetHandlerFactory.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2Rlc3RpbmF0aW9uL0Rlc3RpbmF0aW9uRGF0YXNldEhhbmRsZXJGYWN0b3J5LmphdmE=) | | |
   | ... and [1413 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...a230fb0](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] ZihanLi58 commented on a change in pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on a change in pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#discussion_r681245048



##########
File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -574,19 +592,21 @@ private String sanitizeApplicationId(String applicationId) {
 
   @VisibleForTesting
   Optional<ApplicationId> getReconnectableApplicationId() throws YarnException, IOException {
-    List<ApplicationReport> applicationReports =
-        this.yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES);
-    if (applicationReports == null || applicationReports.isEmpty()) {
-      return Optional.absent();
-    }
+    for (YarnClient yarnClient: potentialYarnClients.values()) {
+      List<ApplicationReport> applicationReports = yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES);
+      if (applicationReports == null || applicationReports.isEmpty()) {
+        continue;
+      }
 
-    // Try to find an application with a matching application name
-    for (ApplicationReport applicationReport : applicationReports) {
-      if (this.applicationName.equals(applicationReport.getName())) {
-        String applicationId = sanitizeApplicationId(applicationReport.getApplicationId().toString());
-        LOGGER.info("Found reconnectable application with application ID: " + applicationId);
-        LOGGER.info("Application Tracking URL: " + applicationReport.getTrackingUrl());
-        return Optional.of(applicationReport.getApplicationId());
+      // Try to find an application with a matching application name
+      for (ApplicationReport applicationReport : applicationReports) {

Review comment:
       yeah, in that case, we will continue to check next potential clusters, and if we found nothing in all clusters, we will return Optional.absent on line 613




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e38ba6) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **increase** coverage by `1.80%`.
   > The diff coverage is `32.50%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   + Coverage     46.54%   48.35%   +1.80%     
   + Complexity    10136     7514    -2622     
   ============================================
     Files          2051     1421     -630     
     Lines         79547    55831   -23716     
     Branches       8880     6428    -2452     
   ============================================
   - Hits          37027    26996   -10031     
   + Misses        39091    26309   -12782     
   + Partials       3429     2526     -903     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `25.40% <32.50%> (+1.09%)` | :arrow_up: |
   | [...blin/data/management/copy/RecursivePathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvUmVjdXJzaXZlUGF0aEZpbmRlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...data/management/copy/extractor/EmptyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0VtcHR5RXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...a/management/copy/hive/HiveLocationDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlTG9jYXRpb25EZXNjcmlwdG9yLmphdmE=) | `36.17% <0.00%> (-14.90%)` | :arrow_down: |
   | [...anagement/copy/hive/UnpartitionedTableFileSet.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9VbnBhcnRpdGlvbmVkVGFibGVGaWxlU2V0LmphdmE=) | `44.44% <0.00%> (-12.97%)` | :arrow_down: |
   | [...ache/gobblin/util/commit/DeleteFileCommitStep.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9jb21taXQvRGVsZXRlRmlsZUNvbW1pdFN0ZXAuamF2YQ==) | `34.04% <0.00%> (-8.52%)` | :arrow_down: |
   | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `52.67% <0.00%> (-8.46%)` | :arrow_down: |
   | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `67.54% <0.00%> (-6.63%)` | :arrow_down: |
   | [...e/gobblin/data/management/copy/hive/HiveUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlVXRpbHMuamF2YQ==) | `46.34% <0.00%> (-4.88%)` | :arrow_down: |
   | ... and [639 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...2e38ba6](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] sv2000 merged pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled

Posted by GitBox <gi...@apache.org>.
sv2000 merged pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (edb046a) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `0.18%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   46.35%   -0.19%     
   + Complexity    10136     3065    -7071     
   ============================================
     Files          2051      644    -1407     
     Lines         79547    25091   -54456     
     Branches       8880     2996    -5884     
   ============================================
   - Hits          37027    11632   -25395     
   + Misses        39091    12206   -26885     
   + Partials       3429     1253    -2176     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
   | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `32.23% <0.00%> (-5.79%)` | :arrow_down: |
   | [...he/gobblin/service/modules/flow/FlowGraphPath.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L0Zsb3dHcmFwaFBhdGguamF2YQ==) | `81.15% <0.00%> (-2.67%)` | :arrow_down: |
   | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `58.06% <0.00%> (-2.16%)` | :arrow_down: |
   | [...in/service/modules/core/GobblinServiceManager.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dvYmJsaW5TZXJ2aWNlTWFuYWdlci5qYXZh) | `50.20% <0.00%> (-0.84%)` | :arrow_down: |
   | [...in/service/modules/orchestration/Orchestrator.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL09yY2hlc3RyYXRvci5qYXZh) | `53.10% <0.00%> (-0.35%)` | :arrow_down: |
   | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `58.59% <0.00%> (-0.27%)` | :arrow_down: |
   | [.../org/apache/gobblin/service/ServiceConfigKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9TZXJ2aWNlQ29uZmlnS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ava/org/apache/gobblin/util/hadoop/TokenUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvaGFkb29wL1Rva2VuVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | ... and [1416 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...edb046a](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] umustafi commented on a change in pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#discussion_r681322550



##########
File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -253,8 +259,16 @@ public GobblinYarnAppLauncher(Config config, YarnConfiguration yarnConfiguration
     YarnHelixUtils.setYarnClassPath(config, this.yarnConfiguration);
     YarnHelixUtils.setAdditionalYarnClassPath(config, this.yarnConfiguration);
     this.yarnConfiguration.set("fs.automatic.close", "false");
-    this.yarnClient = YarnClient.createYarnClient();
-    this.yarnClient.init(this.yarnConfiguration);
+    this.originalYarnRMAddress = this.yarnConfiguration.get(GobblinYarnConfigurationKeys.YARN_RESOURCE_MANAGER_ADDRESS);
+    this.potentialYarnClients = new HashMap();
+    Set<String> potentialRMAddresses = new HashSet<>(ConfigUtils.getStringList(config, GobblinYarnConfigurationKeys.OTHER_YARN_RESOURCE_MANAGER_ADDRESSES));
+    potentialRMAddresses.add(originalYarnRMAddress);
+    for (String rmAddress : potentialRMAddresses) {
+       YarnClient tmpYarnClient = YarnClient.createYarnClient();

Review comment:
       nit but i think the spacing is off here - Alex told me to run "silent code cleanup" in Intellij on blocks of code we change to auto correct these. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a230fb0) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `0.10%`.
   > The diff coverage is `32.50%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   46.44%   -0.11%     
   + Complexity    10136    10121      -15     
   ============================================
     Files          2051     2052       +1     
     Lines         79547    79644      +97     
     Branches       8880     8892      +12     
   ============================================
   - Hits          37027    36989      -38     
   - Misses        39091    39218     +127     
   - Partials       3429     3437       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `25.40% <32.50%> (+1.09%)` | :arrow_up: |
   | [...blin/data/management/copy/RecursivePathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvUmVjdXJzaXZlUGF0aEZpbmRlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...data/management/copy/extractor/EmptyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0VtcHR5RXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...che/gobblin/runtime/TaskStateCollectorService.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlQ29sbGVjdG9yU2VydmljZS5qYXZh) | `63.30% <0.00%> (-15.77%)` | :arrow_down: |
   | [...a/management/copy/hive/HiveLocationDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlTG9jYXRpb25EZXNjcmlwdG9yLmphdmE=) | `36.17% <0.00%> (-14.90%)` | :arrow_down: |
   | [...anagement/copy/hive/UnpartitionedTableFileSet.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9VbnBhcnRpdGlvbmVkVGFibGVGaWxlU2V0LmphdmE=) | `44.44% <0.00%> (-12.97%)` | :arrow_down: |
   | [...ache/gobblin/util/commit/DeleteFileCommitStep.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9jb21taXQvRGVsZXRlRmlsZUNvbW1pdFN0ZXAuamF2YQ==) | `34.04% <0.00%> (-8.52%)` | :arrow_down: |
   | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `52.67% <0.00%> (-8.46%)` | :arrow_down: |
   | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `67.54% <0.00%> (-6.63%)` | :arrow_down: |
   | ... and [29 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...a230fb0](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] sv2000 commented on a change in pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#discussion_r681079489



##########
File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnConfigurationKeys.java
##########
@@ -37,6 +37,8 @@
   public static final int DEFAULT_RELEASED_CONTAINERS_CACHE_EXPIRY_SECS = 300;
   public static final String APP_VIEW_ACL = GOBBLIN_YARN_PREFIX + "appViewAcl";
   public static final String DEFAULT_APP_VIEW_ACL = "*";
+  public static final String YARN_RESOURCEMANAGER_ADDRESS= "yarn.resourcemanager.address";
+  public static final String POTENTIAL_YARN_RESOURCEMANAGER_ADDRESSES= "potential.yarn.resourcemanager.addresses";

Review comment:
       OTHER_YARN_RESOURCE_MANAGER_ADDRESSES instead of POTENTIAL_

##########
File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -574,19 +592,21 @@ private String sanitizeApplicationId(String applicationId) {
 
   @VisibleForTesting
   Optional<ApplicationId> getReconnectableApplicationId() throws YarnException, IOException {
-    List<ApplicationReport> applicationReports =
-        this.yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES);
-    if (applicationReports == null || applicationReports.isEmpty()) {
-      return Optional.absent();
-    }
+    for (YarnClient yarnClient: potentialYarnClients.values()) {
+      List<ApplicationReport> applicationReports = yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES);
+      if (applicationReports == null || applicationReports.isEmpty()) {
+        continue;
+      }
 
-    // Try to find an application with a matching application name
-    for (ApplicationReport applicationReport : applicationReports) {
-      if (this.applicationName.equals(applicationReport.getName())) {
-        String applicationId = sanitizeApplicationId(applicationReport.getApplicationId().toString());
-        LOGGER.info("Found reconnectable application with application ID: " + applicationId);
-        LOGGER.info("Application Tracking URL: " + applicationReport.getTrackingUrl());
-        return Optional.of(applicationReport.getApplicationId());
+      // Try to find an application with a matching application name
+      for (ApplicationReport applicationReport : applicationReports) {

Review comment:
       What happens if applicationReports is null? We won't enter this loop. Shouldn't we return Optional.absent() for that case?

##########
File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -835,14 +855,22 @@ private void setupSecurityTokens(ContainerLaunchContext containerLaunchContext)
 
     TokenUtils.getAllFSTokens(new Configuration(), credentials, renewerName,
         Optional.absent(), ConfigUtils.getStringList(this.config, TokenUtils.OTHER_NAMENODES));
-
+    // Only pass token here and no secrets. (since there is no simple way to remove single token/ get secrets)
+    // For RM token, only pass the RM token for the current RM, or the RM will fail to update the token
+    Credentials finalCredentials = new Credentials();
+    for ( Token<? extends TokenIdentifier> token: credentials.getAllTokens()) {

Review comment:
       Nit: Remove unnecessary whitespaces after "for" and "(".




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e38ba6) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `0.09%`.
   > The diff coverage is `32.50%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   46.45%   -0.10%     
   + Complexity    10136    10115      -21     
   ============================================
     Files          2051     2052       +1     
     Lines         79547    79582      +35     
     Branches       8880     8885       +5     
   ============================================
   - Hits          37027    36966      -61     
   - Misses        39091    39180      +89     
   - Partials       3429     3436       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `25.40% <32.50%> (+1.09%)` | :arrow_up: |
   | [...blin/data/management/copy/RecursivePathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvUmVjdXJzaXZlUGF0aEZpbmRlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...data/management/copy/extractor/EmptyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0VtcHR5RXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...a/management/copy/hive/HiveLocationDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlTG9jYXRpb25EZXNjcmlwdG9yLmphdmE=) | `36.17% <0.00%> (-14.90%)` | :arrow_down: |
   | [...anagement/copy/hive/UnpartitionedTableFileSet.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9VbnBhcnRpdGlvbmVkVGFibGVGaWxlU2V0LmphdmE=) | `44.44% <0.00%> (-12.97%)` | :arrow_down: |
   | [...ache/gobblin/util/commit/DeleteFileCommitStep.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9jb21taXQvRGVsZXRlRmlsZUNvbW1pdFN0ZXAuamF2YQ==) | `34.04% <0.00%> (-8.52%)` | :arrow_down: |
   | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `52.67% <0.00%> (-8.46%)` | :arrow_down: |
   | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `67.54% <0.00%> (-6.63%)` | :arrow_down: |
   | [...e/gobblin/data/management/copy/hive/HiveUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlVXRpbHMuamF2YQ==) | `46.34% <0.00%> (-4.88%)` | :arrow_down: |
   | ... and [19 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...2e38ba6](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a230fb0) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `3.51%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   43.03%   -3.52%     
   + Complexity    10136     1942    -8194     
   ============================================
     Files          2051      394    -1657     
     Lines         79547    16877   -62670     
     Branches       8880     2076    -6804     
   ============================================
   - Hits          37027     7263   -29764     
   + Misses        39091     8815   -30276     
   + Partials       3429      799    -2630     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `58.59% <0.00%> (-0.27%)` | :arrow_down: |
   | [...ava/org/apache/gobblin/util/hadoop/TokenUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvaGFkb29wL1Rva2VuVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.37% <0.00%> (ø)` | |
   | [.../gobblin/util/test/RetentionTestDataGenerator.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC90ZXN0L1JldGVudGlvblRlc3REYXRhR2VuZXJhdG9yLmphdmE=) | | |
   | [...va/org/apache/gobblin/metrics/event/EventName.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9FdmVudE5hbWUuamF2YQ==) | | |
   | [.../converter/BytesToRecordWithMetadataConverter.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0YWRhdGEvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29udmVydGVyL0J5dGVzVG9SZWNvcmRXaXRoTWV0YWRhdGFDb252ZXJ0ZXIuamF2YQ==) | | |
   | [...mpaction/action/CompactionMarkDirectoryAction.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25NYXJrRGlyZWN0b3J5QWN0aW9uLmphdmE=) | | |
   | [...copy/predicates/RegistrationTimeSkipPredicate.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHJlZGljYXRlcy9SZWdpc3RyYXRpb25UaW1lU2tpcFByZWRpY2F0ZS5qYXZh) | | |
   | [...gobblin/runtime/listeners/AbstractJobListener.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbGlzdGVuZXJzL0Fic3RyYWN0Sm9iTGlzdGVuZXIuamF2YQ==) | | |
   | [...lin/source/extractor/extract/EventBasedSource.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL2V4dHJhY3Rvci9leHRyYWN0L0V2ZW50QmFzZWRTb3VyY2UuamF2YQ==) | | |
   | ... and [1649 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...a230fb0](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with federated Yarn cluster enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (edb046a) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `0.11%`.
   > The diff coverage is `32.50%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   46.42%   -0.12%     
   + Complexity    10136    10117      -19     
   ============================================
     Files          2051     2052       +1     
     Lines         79547    79644      +97     
     Branches       8880     8892      +12     
   ============================================
   - Hits          37027    36977      -50     
   - Misses        39091    39229     +138     
   - Partials       3429     3438       +9     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `25.40% <32.50%> (+1.09%)` | :arrow_up: |
   | [...blin/data/management/copy/RecursivePathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvUmVjdXJzaXZlUGF0aEZpbmRlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...data/management/copy/extractor/EmptyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0VtcHR5RXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
   | [...che/gobblin/runtime/TaskStateCollectorService.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlQ29sbGVjdG9yU2VydmljZS5qYXZh) | `63.30% <0.00%> (-15.77%)` | :arrow_down: |
   | [...a/management/copy/hive/HiveLocationDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlTG9jYXRpb25EZXNjcmlwdG9yLmphdmE=) | `36.17% <0.00%> (-14.90%)` | :arrow_down: |
   | [...anagement/copy/hive/UnpartitionedTableFileSet.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9VbnBhcnRpdGlvbmVkVGFibGVGaWxlU2V0LmphdmE=) | `44.44% <0.00%> (-12.97%)` | :arrow_down: |
   | [...ache/gobblin/util/commit/DeleteFileCommitStep.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9jb21taXQvRGVsZXRlRmlsZUNvbW1pdFN0ZXAuamF2YQ==) | `34.04% <0.00%> (-8.52%)` | :arrow_down: |
   | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `52.67% <0.00%> (-8.46%)` | :arrow_down: |
   | ... and [32 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...edb046a](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter commented on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-889542693


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3345](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e38ba6) into [master](https://codecov.io/gh/apache/gobblin/commit/48af4c61c4780740b63931901c756e5a184e763b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48af4c6) will **decrease** coverage by `3.51%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3345/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3345      +/-   ##
   ============================================
   - Coverage     46.54%   43.03%   -3.52%     
   + Complexity    10136     1942    -8194     
   ============================================
     Files          2051      394    -1657     
     Lines         79547    16877   -62670     
     Branches       8880     2076    -6804     
   ============================================
   - Hits          37027     7263   -29764     
   + Misses        39091     8815   -30276     
   + Partials       3429      799    -2630     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `58.59% <0.00%> (-0.27%)` | :arrow_down: |
   | [...ava/org/apache/gobblin/util/hadoop/TokenUtils.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvaGFkb29wL1Rva2VuVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...in/kafka/schemareg/CachingKafkaSchemaRegistry.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3NjaGVtYXJlZy9DYWNoaW5nS2Fma2FTY2hlbWFSZWdpc3RyeS5qYXZh) | | |
   | [.../gobblin/policies/schema/SchemaRowCheckPolicy.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3BvbGljaWVzL3NjaGVtYS9TY2hlbWFSb3dDaGVja1BvbGljeS5qYXZh) | | |
   | [...ache/gobblin/runtime/api/JobExecutionLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvbkxhdW5jaGVyLmphdmE=) | | |
   | [.../copy/extractor/FileAwareInputStreamExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0ZpbGVBd2FyZUlucHV0U3RyZWFtRXh0cmFjdG9yLmphdmE=) | | |
   | [...rg/apache/gobblin/writer/ConsoleWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9Db25zb2xlV3JpdGVyQnVpbGRlci5qYXZh) | | |
   | [...dataset/test/SimpleDatasetPartitionForTesting.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC90ZXN0L1NpbXBsZURhdGFzZXRQYXJ0aXRpb25Gb3JUZXN0aW5nLmphdmE=) | | |
   | [...he/gobblin/runtime/job\_catalog/JobCatalogBase.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSm9iQ2F0YWxvZ0Jhc2UuamF2YQ==) | | |
   | [...ain/java/org/apache/gobblin/runtime/fork/Fork.java](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvZm9yay9Gb3JrLmphdmE=) | | |
   | ... and [1648 more](https://codecov.io/gh/apache/gobblin/pull/3345/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [48af4c6...2e38ba6](https://codecov.io/gh/apache/gobblin/pull/3345?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] sv2000 commented on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
sv2000 commented on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-891138616


   @ZihanLi58 - you may want to replace reference to "Robin" in the PR description with "federated Yarn cluster". 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] ZihanLi58 commented on pull request #3345: [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP)

Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#issuecomment-891322896


   > A few comments. One question: How does token refresh work in a federated Yarn cluster for keytab based authentication? Will it work as is after the changes in this PR?
   
   @sv2000 For original implementation, i.e. there is only one yarn cluster, it should still work after this change. But to support the federated clusters, we may need to implement the method to get delegation token from remote yarn clusters and I don't see anything related to rm token in our keytab based authentication.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org