You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/08/11 04:17:29 UTC
[GitHub] [incubator-uniffle] zhengchenyu opened a new pull request, #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
zhengchenyu opened a new pull request, #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131
### What changes were proposed in this pull request?
For MR/Tez, as applications is stopped, we should unregister shuffle explicitly. There are no need to wait timeout.
### Why are the changes needed?
Fix: #1115
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
test in yarn cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674269787
> Could we add UT for this pr?
I am confused about UT. This PR make effect in main. For now, I did not find a good way to construct UT yet.
I think if integration test pass, there will be no any problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674257207
Could we add UT for this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674196711
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1131](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7fb0f91) into [master](https://app.codecov.io/gh/apache/incubator-uniffle/commit/0bde6ba30f59060e52a0b857c96538780c05db7b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (0bde6ba) will **increase** coverage by `0.89%`.
> Report is 3 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #1131 +/- ##
============================================
+ Coverage 54.27% 55.17% +0.89%
+ Complexity 2561 2093 -468
============================================
Files 388 316 -72
Lines 21985 14090 -7895
Branches 1825 1266 -559
============================================
- Hits 11933 7774 -4159
+ Misses 9348 5898 -3450
+ Partials 704 418 -286
```
| [Files Changed](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50LW1yL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9tYXByZWR1Y2UvdjIvYXBwL1Jzc01SQXBwTWFzdGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [78 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1131/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zhengchenyu commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290940489
##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
rssNmHttpPort = nmHttpPort;
rssContainerID = containerId;
rssContainerAllocator = null;
+ shuffleWriteClient = client;
}
private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
+ @Override
+ protected void serviceStop() throws Exception {
Review Comment:
Yes! In general, this will be invoked explicitly.
Below is the steps:
(1) KillNewJobTransition/CommitSucceededTransition/InternalTerminationTransition => job.finished
(2) In job.finished, send JobFinishEvent
(3) JobFinishEventHandler handle the JobFinishEvent and called shutDownJob
(4) shutDownJob => MRAppMaster.this.stop() => serviceStop
Below is related logs:
```
2023-08-11 14:19:40,033 INFO [Thread-132] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2023-08-11 14:19:40,033 INFO [Thread-132] org.apache.hadoop.mapreduce.v2.app.RssMRAppMaster: Unregister shuffle for app appattempt_1691730663832_0001_000001
2023-08-11 14:19:40,046 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e3969_1691730663832_0001_01_000035
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston merged pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zhengchenyu commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290940489
##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
rssNmHttpPort = nmHttpPort;
rssContainerID = containerId;
rssContainerAllocator = null;
+ shuffleWriteClient = client;
}
private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
+ @Override
+ protected void serviceStop() throws Exception {
Review Comment:
Yes! In general, this will be invoked explicitly.
Below is the steps:
(1) KillNewJobTransition/CommitSucceededTransition/InternalTerminationTransition => job.finished
(2) In job.finished, send JobFinishEvent
(3) JobFinishEventHandler handle the JobFinishEvent and called shutDownJob
(4) shutDownJob => MRAppMaster.this.stop() => serviceStop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#issuecomment-1674299548
> > Could we add UT for this pr?
>
> I am confused about UT. This PR make effect in main. For now, I did not find a good way to construct UT yet. I think if integration test pass, there will be no any problem.
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #1131: [#1115] improvement: Unregister shuffle explicitly when application i…
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1131:
URL: https://github.com/apache/incubator-uniffle/pull/1131#discussion_r1290933900
##########
client-mr/core/src/main/java/org/apache/hadoop/mapreduce/v2/app/RssMRAppMaster.java:
##########
@@ -111,20 +113,32 @@ public RssMRAppMaster(
rssNmHttpPort = nmHttpPort;
rssContainerID = containerId;
rssContainerAllocator = null;
+ shuffleWriteClient = client;
}
private static final Logger LOG = LoggerFactory.getLogger(RssMRAppMaster.class);
+ @Override
+ protected void serviceStop() throws Exception {
Review Comment:
this will be invoked explicitly by MR ?
##########
client-tez/src/main/java/org/apache/tez/dag/app/TezRemoteShuffleManager.java:
##########
@@ -109,6 +109,14 @@ public void start() throws Exception {
@Override
public void shutdown() throws Exception {
Review Comment:
ditto?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org