You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2020/11/24 21:21:00 UTC
[jira] [Issue Comment Deleted] (MAPREDUCE-7309) Improve performance
of reading resource request for mapper/reducers from config
[ https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Bacsko updated MAPREDUCE-7309:
------------------------------------
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m 54s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m 0s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | | {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 9m 37s{color} | [/branch-mvninstall-root.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-mvninstall-root.txt] | {color:red} root in branch-3.1 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s{color} | [/branch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in branch-3.1 failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | [/buildtool-branch-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/buildtool-branch-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:orange} The patch fails to run checkstyle in hadoop-mapreduce-client-app {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 13s{color} | [/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in branch-3.1 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m 55s{color} | [/branch-shadedclient.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-shadedclient.txt] | {color:red} branch has errors when building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s{color} | [/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in branch-3.1 failed. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m 27s{color} | | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 12s{color} | [/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in branch-3.1 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s{color} | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s{color} | [/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s{color} | [/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m 0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 10s{color} | [/buildtool-patch-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/buildtool-patch-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:orange} The patch fails to run checkstyle in hadoop-mapreduce-client-app {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 11s{color} | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m 38s{color} | [/patch-shadedclient.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-shadedclient.txt] | {color:red} patch has errors when building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s{color} | [/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s{color} | [/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s{color} | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt] | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 28s{color} | | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/artifact/out/Dockerfile |
| JIRA Issue | MAPREDUCE-7309 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13015953/MAPREDUCE-7309-branch-3.1-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell |
| uname | Linux 67dcc5808d36 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | branch-3.1 / 4638ed94dbf20e65293e675222e7bdaeb141b68a |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/testReport/ |
| Max. process+thread count | 93 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output | https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/41/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
)
> Improve performance of reading resource request for mapper/reducers from config
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7309
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
> Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
> Reporter: Wangda Tan
> Assignee: Peter Bacsko
> Priority: Major
> Fix For: 3.4.0
>
> Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309-004.patch, MAPREDUCE-7309-005.patch, MAPREDUCE-7309-branch-3.1-001.patch, MAPREDUCE-7309-branch-3.2-001.patch, MAPREDUCE-7309-branch-3.3-001.patch, MAPREDUCE-7309.001.patch, MAPREDUCE-7309.002.patch
>
>
> This is an issue could affect all the releases which includes YARN-6927.
> Basically, we use regex match repeatedly when we read mapper/reducer resource request from config files. When we have large config file, and large number of splits, it could take a long time.
> We saw AM could take hours to parse config when we have 200k+ splits, with a large config file (hundreds of kbs).
> The problamtic part is this:
> {noformat}
> private void populateResourceCapability(TaskType taskType) {
> String resourceTypePrefix =
> getResourceTypePrefix(taskType);
> boolean memorySet = false;
> boolean cpuVcoresSet = false;
> if (resourceTypePrefix != null) {
> List<ResourceInformation> resourceRequests =
> ResourceUtils.getRequestedResourcesFromConfig(conf,
> resourceTypePrefix);
> {noformat}
> Inside {{ResourceUtils.getRequestedResourcesFromConfig()}}, we call {{Configuration.getValByRegex()}} which goes through all property keys that come from the MapReduce job configuration (jobconf.xml). If the job config is large (eg. due to being part of an MR pipeline and it was populated by an earlier job), then this results in running a regexp match unnecessarily for all properties over and over again. This is not necessary, because all mappers and reducers will have the same config, respectively.
> We should do proper caching for pre-configured resource requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org