You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/01/11 16:14:27 UTC
[jira] Commented: (HADOOP-875) memory model is not accurate enough
for map side sorts
[ https://issues.apache.org/jira/browse/HADOOP-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463912 ]
Devaraj Das commented on HADOOP-875:
------------------------------------
Could you please let me know whether there were multiple spills-to-disk of map outputs for those failed maps?
> memory model is not accurate enough for map side sorts
> ------------------------------------------------------
>
> Key: HADOOP-875
> URL: https://issues.apache.org/jira/browse/HADOOP-875
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
>
> I configured a sort (IdentityMapper) with large compressed values with the map output buffer (io.sort.mb) set to 300mb and the child jvm heap size set to 900mb and some of the maps were running out of memory deterministically. I suspect that some part of the data path is not handling large values well and is consuming large amounts of ram.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira