You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Wangda Tan (Jira)" <ji...@apache.org> on 2020/11/20 19:58:00 UTC
[jira] [Assigned] (MAPREDUCE-7309) Improve performance of reading
resource request for mapper/reducers from config
[ https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wangda Tan reassigned MAPREDUCE-7309:
-------------------------------------
Assignee: Wangda Tan
> Improve performance of reading resource request for mapper/reducers from config
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7309
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
> Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Major
> Attachments: MAPREDUCE-7309.001.patch
>
>
> This is an issue could affect all the releases which includes YARN-6927.
> Basically, we use regex match repeatly when we read mapper/reducer resource request from config files. When we have large config file, and large number of splits, it could take a long time.
> We saw AM could take hours to parse config when we have 200k+ splits, with a large config file (hundreds of kbs).
> We should do proper caching for pre-configured resource requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org