You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/01/09 07:19:27 UTC
[jira] Created: (HADOOP-868) Fix the merge method on Maps to limit
the number of open files
Fix the merge method on Maps to limit the number of open files
--------------------------------------------------------------
Key: HADOOP-868
URL: https://issues.apache.org/jira/browse/HADOOP-868
Project: Hadoop
Issue Type: Bug
Components: mapred
Reporter: Devaraj Das
Assigned To: Devaraj Das
The mergeParts method should not hardcode the factor for the merge and instead rely on the configured default. Also, there is an unnecessary open call made in the method.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-868) Fix the merge method on Maps to limit
the number of open files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-868:
-------------------------------
Status: Patch Available (was: Open)
The patch fixes the following:
1) Removes a redundant file "open" method call from mergeParts
2) All index files are not kept open at once. Now, the index file for a spill is opened, the offsets are read after "seek"ing to the correct place, and then the file is immediately closed.
3) The hardcoding of merge factor to the number of spills is not done; merge factor used for the merges is the default (configured via io.sort.factor).
> Fix the merge method on Maps to limit the number of open files
> --------------------------------------------------------------
>
> Key: HADOOP-868
> URL: https://issues.apache.org/jira/browse/HADOOP-868
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Assigned To: Devaraj Das
> Attachments: 868.patch
>
>
> The mergeParts method should not hardcode the factor for the merge and instead rely on the configured default. Also, there is an unnecessary open call made in the method.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-868) Fix the merge method on Maps to limit
the number of open files
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-868:
-------------------------------
Attachment: 868.patch
> Fix the merge method on Maps to limit the number of open files
> --------------------------------------------------------------
>
> Key: HADOOP-868
> URL: https://issues.apache.org/jira/browse/HADOOP-868
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Assigned To: Devaraj Das
> Attachments: 868.patch
>
>
> The mergeParts method should not hardcode the factor for the merge and instead rely on the configured default. Also, there is an unnecessary open call made in the method.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-868) Fix the merge method on Maps to
limit the number of open files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463231 ]
Hadoop QA commented on HADOOP-868:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12348535/868.patch applied and successfully tested against trunk revision r494253.
> Fix the merge method on Maps to limit the number of open files
> --------------------------------------------------------------
>
> Key: HADOOP-868
> URL: https://issues.apache.org/jira/browse/HADOOP-868
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Assigned To: Devaraj Das
> Attachments: 868.patch
>
>
> The mergeParts method should not hardcode the factor for the merge and instead rely on the configured default. Also, there is an unnecessary open call made in the method.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-868) Fix the merge method on Maps to limit
the number of open files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-868:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.10.1
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Devaraj!
> Fix the merge method on Maps to limit the number of open files
> --------------------------------------------------------------
>
> Key: HADOOP-868
> URL: https://issues.apache.org/jira/browse/HADOOP-868
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Assigned To: Devaraj Das
> Fix For: 0.10.1
>
> Attachments: 868.patch
>
>
> The mergeParts method should not hardcode the factor for the merge and instead rely on the configured default. Also, there is an unnecessary open call made in the method.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira