You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/08/11 04:17:29 UTC

[GitHub] [incubator-uniffle] zhengchenyu opened a new pull request, #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

zhengchenyu opened a new pull request, #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131

   
   ### What changes were proposed in this pull request?
   
   For MR/Tez, as applications is stopped, we should unregister shuffle explicitly. There are no need to wait timeout.
   
   ### Why are the changes needed?
   
   Fix: #1115 
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   test in yarn cluster
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674269787

   > Could we add UT for this pr?
   
   I am confused about UT. This PR make effect in main. For now, I did not find a good way to construct UT yet.
   I think if integration test pass, there will be no any problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674257207

   Could we add UT for this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674196711

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#1131](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7fb0f91) into [master](https://app.codecov.io/gh/apache/incubator-uniffle/commit/0bde6ba30f59060e52a0b857c96538780c05db7b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (0bde6ba) will **increase** coverage by `0.89%`.
   > Report is 3 commits behind head on master.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1131      +/-   ##
   ============================================
   + Coverage     54.27%   55.17%   +0.89%     
   + Complexity     2561     2093     -468     
   ============================================
     Files           388      316      -72     
     Lines         21985    14090    -7895     
     Branches       1825     1266     -559     
   ============================================
   - Hits          11933     7774    -4159     
   + Misses         9348     5898    -3450     
   + Partials        704      418     -286     
   ```
   
   
   | [Files Changed](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50LW1yL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9tYXByZWR1Y2UvdjIvYXBwL1Jzc01SQXBwTWFzdGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   
   ... and [78 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290940489


##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
     rssNmHttpPort = nmHttpPort;
     rssContainerID = containerId;
     rssContainerAllocator = null;
+    shuffleWriteClient = client;
   }
 
   private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
 
+  @Override
+  protected void serviceStop() throws Exception {

Review Comment:
   Yes! In general, this will be invoked explicitly.
   Below is the steps:
   (1) KillNewJobTransition/CommitSucceededTransition/InternalTerminationTransition => job.finished 
   (2) In job.finished, send JobFinishEvent
   (3) JobFinishEventHandler handle the JobFinishEvent and called shutDownJob
   (4) shutDownJob => MRAppMaster.this.stop() => serviceStop
   
   Below is related logs:
   
   ```
   2023-08-11 14:19:40,033 INFO [Thread-132] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
   2023-08-11 14:19:40,033 INFO [Thread-132] org.apache.hadoop.mapreduce.v2.app.RssMRAppMaster: Unregister shuffle for app appattempt_1691730663832_0001_000001
   2023-08-11 14:19:40,046 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e3969_1691730663832_0001_01_000035
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston merged pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290940489


##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
     rssNmHttpPort = nmHttpPort;
     rssContainerID = containerId;
     rssContainerAllocator = null;
+    shuffleWriteClient = client;
   }
 
   private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
 
+  @Override
+  protected void serviceStop() throws Exception {

Review Comment:
   Yes! In general, this will be invoked explicitly.
   Below is the steps:
   (1) KillNewJobTransition/CommitSucceededTransition/InternalTerminationTransition => job.finished 
   (2) In job.finished, send JobFinishEvent
   (3) JobFinishEventHandler handle the JobFinishEvent and called shutDownJob
   (4) shutDownJob => MRAppMaster.this.stop() => serviceStop



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674299548

   > > Could we add UT for this pr?
   > 
   > I am confused about UT. This PR make effect in main. For now, I did not find a good way to construct UT yet. I think if integration test pass, there will be no any problem.
   
   OK.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290933900


##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
     rssNmHttpPort = nmHttpPort;
     rssContainerID = containerId;
     rssContainerAllocator = null;
+    shuffleWriteClient = client;
   }
 
   private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
 
+  @Override
+  protected void serviceStop() throws Exception {

Review Comment:
   this will be invoked explicitly by MR ?



##########
client-tez/src/main/java/org/apache/tez/dag/app/TezRemoteShuffleManager.java:
##########
@@ -109,6 +109,14 @@ public void start() throws Exception {
 
   @Override
   public void shutdown() throws Exception {

Review Comment:
   ditto?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org