You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "lewismc (via GitHub)" <gi...@apache.org> on 2026/03/09 13:33:49 UTC
[PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
lewismc opened a new pull request, #8331:
URL: https://github.com/apache/hadoop/pull/8331
### Description of PR
See [HADOOP-19835](https://issues.apache.org/jira/browse/HADOOP-19835)
### How was this patch tested?
Tested with Hadoop 3.4.3 and Celeborn 0.6.2 and Nutch 1.23-SNAPSHOT. Nutch MapReduce smoke tests were run.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7538. Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3336342503
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
+ MRJobConfig.APPLICATION_MASTER_CLASS);
+ vargs.add(amClass);
Review Comment:
Would you want to do any preflight checking here?
we can't look for the class on the CP as it may arrive in the referenced jar, but should fail fast if the classname is null/""
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
use getTrimmed() to trim whitespace
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
I concur with everything @K0K0V0K says.
- public string constant
- updated default file
- doc change.
Test wise,
1. Create a subclass of the normal MR AM which does something observable in startup (touch a file on a configured path?). Launch a minimal MR job and assert the file exists.
2. Launch an MR job with a configuration that refers to a class which isn't present. Expect failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918255092
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
if so, this is a hack of the MR framework ... I think we should make the AM class configurable as you proposed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2909381782
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Thanks for looking at this patch @pan3793 , I have no issue changing it to `yarn.app.mapreduce.am.class`. I'd really like to hear from @RexXiong as to how Celeborn has been integrated without this change as I could only get it to work with this patched YARNRunner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2933775126
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
@RexXiong thanks for the feedback. Can you please provide guidance on expanding this PR? Anything in addition to
1. changing `yarn.app.mapreduce.am` --> `yarn.app.mapreduce.am.class`, and
2. changing the constant, docs, default config file, etc... how do I do this?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2909137154
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
should it be "yarn.app.mapreduce.am.class" ?
adding a new config requires changing the constant, docs, default config file, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3247356650
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
> 2. changing the constant, docs, default config file, etc... how do I do this?
The constant should be added to **MRJobConfig.class**. As far as I know, it does not have a corresponding default value.
If you modify **MRConfig.class**, you should also update **mapred-default.xml** and **TaskLevelSecurityEnforcement.md**.
I think providing a proper unit test could be quite challenging, since the class loader behavior cannot be fully tested easily. However, adding a new test under **TestYARNRunner.class** would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3246813391
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
For me, the proposal looks good overall.
Since we already have **mapreduce.job.map.class** and **mapreduce.job.reduce.class**, I would suggest naming the new configuration property **mapreduce.job.am.class** for consistency.
May I also ask which Java versions were tested with this change? I was initially wondering whether the module system could cause any issues on newer JDKs, but I think this approach should work fine.
FYI, while I am not fully aware of the intended deployment use case, I believe there are at least two possible mechanisms for making the class available on the NodeManagers:
* Bundling it directly with the application
* Adding it to the MapReduce framework tarball via `org.apache.hadoop.mapred.uploader.FrameworkUploader`
Could I also ask you to add the new property here as well?
https://github.com/apache/hadoop/blob/22008a241a81a5f0a77cf55ead3546502c71df99/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java#L167
That would provide at least some limited control over what kinds of classes can be loaded through this mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918089016
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
A simple example command to reproduce and test
```
docker compose -f docker-compose.yml -f docker-compose.celeborn.yml exec -u hadoop namenode hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.3.jar pi 2 4
```
If you need access to the Docker composition to test, please let me know. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3246813391
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Thanks @lewismc for the improvement!
For me, the proposal looks good overall.
Since we already have **mapreduce.job.map.class** and **mapreduce.job.reduce.class**, I would suggest naming the new configuration property **mapreduce.job.am.class** for consistency.
May I also ask which Java versions were tested with this change? I was initially wondering whether the module system could cause any issues on newer JDKs, but I think this approach should work fine.
FYI, while I am not fully aware of the intended deployment use case, I believe there are at least two possible mechanisms for making the class available on the NodeManagers:
* Bundling it directly with the application
* Adding it to the MapReduce framework tarball via `org.apache.hadoop.mapred.uploader.FrameworkUploader`
Could I also ask you to add the new property here as well?
https://github.com/apache/hadoop/blob/22008a241a81a5f0a77cf55ead3546502c71df99/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java#L167
That would provide at least some limited control over what kinds of classes can be loaded through this mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#issuecomment-4439613523
ping @steveloughran @K0K0V0K because you might be the only one who made changes in the MR module recently. Do you have concerns about making AM class configurable? the proposed config name is `yarn.app.mapreduce.am.class`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#issuecomment-4028635845
cc @RexXiong, how was MR on Celeborn used without this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918077873
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Hi @RexXiong thanks.
I wasn't able to get that working because of how Hadoop builds the AM container command and what yarn.app.mapreduce.am.command-opts is used for.
# How the AM is launched
In `YARNRunner`, the command for the AM container is built in two separate steps:
1. Main class – One place in code does:
* `vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);` So the main class is always `org.apache.hadoop.mapreduce.v2.app.MRAppMaster`. That value is hardcoded; no config key is read for it.
2. Command opts – `yarn.app.mapreduce.am.command-opts` is used elsewhere for JVM options or extra arguments. Those are merged into the same command, but they are not used as the main class. So they end up either:
* as JVM args (e.g. `-Xmx...`), or
* as arguments passed to the main class (i.e. to `MRAppMaster`).
So the actual process looks like:
`java [options from command-opts] org.apache.hadoop.mapreduce.v2.app.MRAppMaster [any extra args] 1>... 2>...`
If you set:
`-Dyarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn`
then that string is treated as part of “options” or “extra args”. It does not replace the main class, so:
* JVM still runs `MRAppMaster` as main.
* `MRAppMasterWithCeleborn` is at best an argument to `MRAppMaster`, not the entry point.
The JVM never executes `MRAppMasterWithCeleborn` as the main class; it always runs `MRAppMaster`. I wasn't able to get the example from the Celeborn doc’s method running without this patch. The main class is fixed in code, and command-opts never controls it.
Unless I am mistaken, to actually run Celeborn’s AM, the main class in that launch command must be `MRAppMasterWithCeleborn`. The only way to do that with the current design is to change the code that builds the command so it takes the main class from config (e.g. `yarn.app.mapreduce.am` or as proposed by @pan3793 `yarn.app.mapreduce.am.class`) instead of always using `APPLICATION_MASTER_CLASS`. That’s what this patch for `YARNRunner` does; command-opts alone can’t do it.
Thanks for any feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918077873
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Hi @RexXiong thanks.
I wasn't able to get that working because of how Hadoop builds the AM container command and what yarn.app.mapreduce.am.command-opts is used for.
# How the AM is launched
In `YARNRunner`, the command for the AM container is built in two separate steps:
1. Main class – One place in code does:
* `vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);` So the main class is always `org.apache.hadoop.mapreduce.v2.app.MRAppMaster`. That value is hardcoded; no config key is read for it.
2. Command opts – `yarn.app.mapreduce.am.command-opts` is used elsewhere for JVM options or extra arguments. Those are merged into the same command, but they are not used as the main class. So they end up either:
* as JVM args (e.g. `-Xmx...`), or
* as arguments passed to the main class (i.e. to `MRAppMaster`).
So the actual process looks like:
`java [options from command-opts] org.apache.hadoop.mapreduce.v2.app.MRAppMaster [any extra args] 1>... 2>...`
If you set:
`-Dyarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn`
then that string is treated as part of “options” or “extra args”. It does not replace the main class, so:
* JVM still runs `MRAppMaster` as main.
* `MRAppMasterWithCeleborn` is at best an argument to `MRAppMaster`, not the entry point.
The JVM never executes `MRAppMasterWithCeleborn` as the main class; it always runs `MRAppMaster`. I wasn't able to get the example from the Celeborn doc’s method running without this patch. The main class is fixed in code, and command-opts never controls it.
Unless I am mistaken, to actually run Celeborn’s AM, the main class in that launch command must be `MRAppMasterWithCeleborn`. The only way to do that with the current design is to change the code that builds the command so it takes the main class from config (e.g. `yarn.app.mapreduce.am` or as proposed by @pan3793 `yarn.app.mapreduce.am.class`) instead of always using `APPLICATION_MASTER_CLASS`. That’s what this patch for `YARNRunner` does; command-opts alone can’t do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7538. Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#issuecomment-4477686576
@steveloughran, thanks for your input. I moved the Jira to MAPREDUCE-7538.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3252504505
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
@K0K0V0K thanks for your input, this sounds promising. @lewismc, do you want to continue this work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2922092223
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
@lewismc Agree with your proposal, make AM class configurable seems more reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "lewismc (via GitHub)" <gi...@apache.org>.
lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918089016
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
BTW, the command I am trying to run
```
docker compose -f docker-compose.yml -f docker-compose.celeborn.yml exec -u hadoop namenode hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.3.jar pi 2 4
```
If you need access to the Docker composition to test, please let me know. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#issuecomment-4030733686
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 1m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 114m 10s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 228m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8331/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4cce92c76378 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8f3cfd1429ab21931db3d6e8b59f44208c29e99c |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8331/1/testReport/ |
| Max. process+thread count | 1258 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8331/1/console |
| versions | git=2.43.0 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2916527255
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Seems the Celeborn document presents another method: `-Dyarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn`, see https://github.com/apache/celeborn?tab=readme-ov-file#deploy-mapreduce-client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918246742
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
```
$ export CLASSPATH=...
$ java -Xmx1g HackMain Main foo bar
```
@lewismc, I guess in the above command, `HackMain` will run as the entrypoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3247356650
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
> 2. changing the constant, docs, default config file, etc... how do I do this?
The constant should be added to **MRJobConfig.class**. As far as I know, it does not have a corresponding default xml.
If you modify **MRConfig.class**, you should also update **mapred-default.xml** and **TaskLevelSecurityEnforcement.md**.
I think providing a proper unit test could be quite challenging, since the class loader behavior cannot be fully tested easily. However, adding a new test under **TestYARNRunner.class** would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19835 Make MapReduce Application Master class configurable in YARNRunner [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#issuecomment-4477426208
I have no problem but
* move jira to MAPREDUCE
* log the value at debug before launching
* test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org