You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/05/25 07:23:55 UTC
[jira] Created: (HADOOP-3443) map outputs should not be renamed
between partitions
map outputs should not be renamed between partitions
----------------------------------------------------
Key: HADOOP-3443
URL: https://issues.apache.org/jira/browse/HADOOP-3443
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.17.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Priority: Critical
If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601930#action_12601930 ]
Hudson commented on HADOOP-3443:
--------------------------------
Integrated in Hadoop-trunk #511 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/])
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: hadoop-3443-1.patch, hadoop-3443-1v17.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3443:
----------------------------------
Status: Patch Available (was: Open)
Should this be fixed in 0.17?
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Attachments: hadoop-3443-1.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3443:
----------------------------------
Attachment: hadoop-3443-1.patch
This patch uses the same directory for the final outputs in the case that there is a single spill. It also adds some logging so you can understand if (and why) the mapper's sort spilled.
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Attachments: hadoop-3443-1.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3443:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.18.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Owen
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: hadoop-3443-1.patch, hadoop-3443-1v17.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600980#action_12600980 ]
Chris Douglas commented on HADOOP-3443:
---------------------------------------
+1 Looks good. Some of the logging could be at the debug level, particularly the logging in the synchronized section of BlockingBuffer.Buffer::write, but it shouldn't be a real issue.
bq. Is it a regression?
No; it looks pretty ancient, but if there's a 0.17.1, including this would only make things better.
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Attachments: hadoop-3443-1.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599886#action_12599886 ]
Doug Cutting commented on HADOOP-3443:
--------------------------------------
> Should this be fixed in 0.17?
Is it a regression?
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Attachments: hadoop-3443-1.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3443) map outputs should not be renamed
between partitions
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3443:
----------------------------------
Attachment: hadoop-3443-1v17.patch
Port to 0.17, if we need it.
> map outputs should not be renamed between partitions
> ----------------------------------------------------
>
> Key: HADOOP-3443
> URL: https://issues.apache.org/jira/browse/HADOOP-3443
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: hadoop-3443-1.patch, hadoop-3443-1v17.patch
>
>
> If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.