You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Zhang Wei (JIRA)" <ji...@apache.org> on 2015/03/20 04:34:38 UTC

[jira] [Created] (MAPREDUCE-6283) MRHistoryServer log files management optimization

Zhang Wei created MAPREDUCE-6283:
------------------------------------

             Summary: MRHistoryServer log files management optimization
                 Key: MAPREDUCE-6283
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6283
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobhistoryserver
            Reporter: Zhang Wei
            Assignee: Zhang Wei
            Priority: Minor


In some heavy computation clusters, there might be a potential hdfs small files problem. The continually submitted MR jobs will create millions of log files (include the application master logs and application logs which are aggregated to hdfs by nodemanager). This optimization design helps to reduce the numbers of log files by merging them into bigger ones.

Get the details from the design doc which I will upload later.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)