You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2024/06/12 12:09:24 UTC
[PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
SteNicholas opened a new pull request, #2558:
URL: https://github.com/apache/celeborn/pull/2558
### What changes were proposed in this pull request?
`MRAppMasterWithCeleborn` sets `mapreduce.celeborn.master.endpoints` via environment variable `CELEBORN_MASTER_ENDPOINTS`. Meanwhile, the default behavior of `MRAppMasterWithCeleborn` adds:
- Sets `mapreduce.job.map.output.collector.class` to `org.apache.hadoop.mapred.CelebornMapOutputCollector`.
- Sets `mapreduce.job.reduce.shuffle.consumer.plugin.class` to `org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumer`.
### Why are the changes needed?
`MRAppMasterWithCeleborn` sets `mapreduce.celeborn.master.endpoints` via `${HADOOP_CONF_DIR}/mapred-site.xml` or `-Dmapreduce.celeborn.master.endpoints` at present. It could not set `mapreduce.celeborn.master.endpoints` by above way for integration with `RMProxy`. It's recommended that `MRAppMasterWithCeleborn` supports setting `mapreduce.celeborn.master.endpoints` via environment variable `CELEBORN_MASTER_ENDPOINTS`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`WordCountTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #2558: [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS
URL: https://github.com/apache/celeborn/pull/2558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2558:
URL: https://github.com/apache/celeborn/pull/2558#discussion_r1638004611
##########
client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java:
##########
@@ -197,5 +199,14 @@ public static void checkJobConf(JobConf conf) {
// Make sure reduces are scheduled only after all map are completed.
conf.setFloat(MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART, 1.0f);
}
+ String masterEndpointsKey = HadoopUtils.MR_PREFIX + CelebornConf.MASTER_ENDPOINTS().key();
+ String masterEndpointsVal = conf.get(masterEndpointsKey);
+ if (masterEndpointsVal == null || masterEndpointsVal.isEmpty()) {
+ logger.info(
+ "MRAppMaster sets {} via environment variable {}.",
+ masterEndpointsKey,
+ MASTER_ENDPOINTS_ENV);
+ conf.set(masterEndpointsKey, ensureGetSysEnv(MASTER_ENDPOINTS_ENV));
Review Comment:
@cxzl25, `ensureGetSysEnv` could throw exception when `CELEBORN_MASTER_ENDPOINTS` is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #2558:
URL: https://github.com/apache/celeborn/pull/2558#discussion_r1637974262
##########
client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java:
##########
@@ -197,5 +199,14 @@ public static void checkJobConf(JobConf conf) {
// Make sure reduces are scheduled only after all map are completed.
conf.setFloat(MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART, 1.0f);
}
+ String masterEndpointsKey = HadoopUtils.MR_PREFIX + CelebornConf.MASTER_ENDPOINTS().key();
+ String masterEndpointsVal = conf.get(masterEndpointsKey);
+ if (masterEndpointsVal == null || masterEndpointsVal.isEmpty()) {
+ logger.info(
+ "MRAppMaster sets {} via environment variable {}.",
+ masterEndpointsKey,
+ MASTER_ENDPOINTS_ENV);
+ conf.set(masterEndpointsKey, ensureGetSysEnv(MASTER_ENDPOINTS_ENV));
Review Comment:
If the user does not set `mapreduce.celeborn.master.endpoints` and `CELEBORN_MASTER_ENDPOINTS`,
Is it possible to get output like this and get env is null exception?
```
MRAppMaster sets mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS.
CELEBORN_MASTER_ENDPOINTS is null
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2558:
URL: https://github.com/apache/celeborn/pull/2558#issuecomment-2162881419
## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/2558?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 40.49%. Comparing base [(`8e2fe74`)](https://app.codecov.io/gh/apache/celeborn/commit/8e2fe74a60a690911d07db6744b167d66e4c45d1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`2f5b58a`)](https://app.codecov.io/gh/apache/celeborn/commit/2f5b58a2a8507927a3e39e9d258f0853c59cde8a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2558 +/- ##
==========================================
+ Coverage 40.47% 40.49% +0.02%
==========================================
Files 222 222
Lines 14286 14286
Branches 1290 1290
==========================================
+ Hits 5781 5783 +2
+ Misses 8173 8171 -2
Partials 332 332
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/2558?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1460] MRAppMasterWithCeleborn supports setting mapreduce.celeborn.master.endpoints via environment variable CELEBORN_MASTER_ENDPOINTS [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #2558:
URL: https://github.com/apache/celeborn/pull/2558#issuecomment-2165378198
Ping @FMX.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org