You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2008/04/10 08:38:04 UTC
[jira] Created: (HADOOP-3226) Run combiner when merging spills from
map output
Run combiner when merging spills from map output
------------------------------------------------
Key: HADOOP-3226
URL: https://issues.apache.org/jira/browse/HADOOP-3226
Project: Hadoop Core
Issue Type: Improvement
Reporter: Chris Douglas
Fix For: 0.18.0
When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3226:
------------------------------------
Release Note: Added run of the combiner on the map side merge if there are more than min.num.spills.for.combine (3 by default), and run of the combiner on the reduce-side spills. (was: Runs the combiner on the map side merge if there are more than min.num.spills.for.combine (3 by default). Also runs the combiner on the reduce-side spills. )
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Open (was: Patch Available)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Open (was: Patch Available)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590134#action_12590134 ]
Hadoop QA commented on HADOOP-3226:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379803/3226-0.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2265/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2265/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2265/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2265/console
This message is automatically generated.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Patch Available (was: Open)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594734#action_12594734 ]
Chris Douglas commented on HADOOP-3226:
---------------------------------------
No tests are updated or included, as the existing tests will verify correctness of the results and the new functionality is both difficult to test and deviations from it are not necessarily incorrect.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: 3226-0.patch
This patch is a first pass. It simply takes the RawValueIterator from the merge and runs the combiner.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593112#action_12593112 ]
Owen O'Malley commented on HADOOP-3226:
---------------------------------------
I think we should get rid of the spill versus merge combiner input/output record counters. I think it would just confuse users over the distinction between them.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591572#action_12591572 ]
Hadoop QA commented on HADOOP-3226:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380717/3226-1.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2303/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2303/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2303/console
This message is automatically generated.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: 3226-1.patch
This patch adds a run of the combiner to the reduce-side spills. It also runs the combiner on the map side merge if there are more than min.num.spills.for.combine (6 by default).
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: 3226-3.patch
Removed separate merge counters and now-unnecessary CombineOutputCollector::setCounter()
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: (was: 3226-0.patch)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3226) Run combiner when
merging spills from map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591403#action_12591403 ]
chris.douglas edited comment on HADOOP-3226 at 4/22/08 11:49 AM:
-----------------------------------------------------------------
This patch adds a run of the combiner to the reduce-side spills. It also runs the combiner on the map side merge if there are more than min.num.spills.for.combine (6 by default). It adds no new test cases because it changes no behavior and should be covered by existing mapred test cases.
was (Author: chris.douglas):
This patch adds a run of the combiner to the reduce-side spills. It also runs the combiner on the map side merge if there are more than min.num.spills.for.combine (6 by default).
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3226:
--------------------------------
Release Note: Runs the combiner on the map side merge if there are more than min.num.spills.for.combine (3 by default). Also runs the combiner on the reduce-side spills.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595209#action_12595209 ]
Hudson commented on HADOOP-3226:
--------------------------------
Integrated in Hadoop-trunk #484 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/484/])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3226:
----------------------------------
Description:
When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
was:When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
Release Note:
Previously, the combiner was run exactly once on each record by the map task. Now, the combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged.
The old semantics are available in Hadoop 0.18 if the user calls:
job.setCombineOnlyOnce(true);
was:Previously, the combiner was run exactly once on each record by the map task. Now, the combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3226:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this with a few fixes for trunk breakage. Thanks, Chris!
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3226:
----------------------------------
Release Note: Previously, the combiner was run exactly once on each record by the map task. Now, the combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. (was: Added run of the combiner on the map side merge if there are more than min.num.spills.for.combine (3 by default), and run of the combiner on the reduce-side spills. )
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: 3226-2.patch
After talking with Owen, changed the default min number of spills from 6 to 3
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592257#action_12592257 ]
Hadoop QA commented on HADOOP-3226:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380884/3226-2.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2321/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2321/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2321/console
This message is automatically generated.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Patch Available (was: Open)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3226:
------------------------------------
Release Note:
Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls:
job.setCombineOnlyOnce(true);
was:
Previously, the combiner was run exactly once on each record by the map task. Now, the combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged.
The old semantics are available in Hadoop 0.18 if the user calls:
job.setCombineOnlyOnce(true);
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587510#action_12587510 ]
Owen O'Malley commented on HADOOP-3226:
---------------------------------------
Either in this patch or a similar one, we should also run the combiner on the reduce input merge spills.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593157#action_12593157 ]
Hadoop QA commented on HADOOP-3226:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381141/3226-3.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2342/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2342/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2342/console
This message is automatically generated.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3226:
----------------------------------
Status: Open (was: Patch Available)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Attachment: 3226-0.patch
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3226) Run combiner when merging spills
from map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-3226:
-------------------------------------
Assignee: Chris Douglas
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Patch Available (was: Open)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Patch Available (was: Open)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3226:
----------------------------------
Status: Patch Available (was: Open)
resubmitting to hudson.
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3226) Run combiner when merging spills from
map output
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3226:
----------------------------------
Status: Open (was: Patch Available)
> Run combiner when merging spills from map output
> ------------------------------------------------
>
> Key: HADOOP-3226
> URL: https://issues.apache.org/jira/browse/HADOOP-3226
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Chris Douglas
> Fix For: 0.18.0
>
> Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.