You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2015/05/06 05:26:16 UTC
[jira] [Updated] (MAPREDUCE-5605) Memory-centric MapReduce aiming
to solve the I/O bottleneck
[ https://issues.apache.org/jira/browse/MAPREDUCE-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-5605:
----------------------------------------
Labels: BB2015-05-TBR (was: )
> Memory-centric MapReduce aiming to solve the I/O bottleneck
> -----------------------------------------------------------
>
> Key: MAPREDUCE-5605
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5605
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.1
> Environment: x86-64 Linux/Unix
> 64-bit jdk7 preferred
> Reporter: Ming Chen
> Assignee: Ming Chen
> Labels: BB2015-05-TBR
> Fix For: 1.0.1
>
> Attachments: MAPREDUCE-5605-v1.patch, TR-mammoth-HUST.pdf, hadoop-core-1.0.1-mammoth-0.9.0.jar
>
>
> Memory is a very important resource to bridge the gap between CPUs and I/O devices. So the idea is to maximize the usage of memory to solve the problem of I/O bottleneck. We developed a multi-threaded task execution engine, which runs in a single JVM on a node. In the execution engine, we have implemented the algorithm of memory scheduling to realize global memory management, based on which we further developed the techniques such as sequential disk accessing, multi-cache and solved the problem of full garbage collection in the JVM. The benchmark results shows that it can get impressive improvement in typical cases. When the a system is relatively short of memory (eg, HPC, small- and medium-size enterprises), the improvement will be even more impressive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)