You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2008/06/18 00:28:44 UTC
[jira] Created: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
keep combiner backward compatible with earlier versions of hadoop
-----------------------------------------------------------------
Key: HADOOP-3586
URL: https://issues.apache.org/jira/browse/HADOOP-3586
Project: Hadoop Core
Issue Type: Bug
Reporter: Olga Natkovich
Fix For: 0.18.0
In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-3586) keep combiner backward compatible with earlier versions of hadoop
Posted by arkady borkovsky <ar...@yahoo-inc.com>.
For my education, can someone tell:
was "combiner guaranteed to run once and only once for each map" a
documented part of the contract ?
-- ab
On Jun 17, 2008, at 3:28 PM, Olga Natkovich (JIRA) wrote:
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/
> HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Fix For: 0.18.0
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once
> and only once for each map. In 17 this compatibility was slightly
> broken: the combiner does not run if a single <K,V> occupies the
> entire sort buffer. In 18, this is further changed to where the
> combiner can be called multiple times on both map and reduce sides.
>
> This breaks Pig's current implementation of the combiner and it is
> not easy to fix in a short period of time.
>
> We would like to ask that for a way for an application to ask for a
> backward compatible behavior for some period of time until it can
> adjust to the new behavior.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606239#action_12606239 ]
Hadoop QA commented on HADOOP-3586:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12384244/3586-2.patch
against trunk revision 669343.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2690/console
This message is automatically generated.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3586:
--------------------------------
Component/s: mapred
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Status: Patch Available (was: Open)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3586) keep combiner backward
compatible with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606135#action_12606135 ]
chris.douglas edited comment on HADOOP-3586 at 6/18/08 4:17 PM:
----------------------------------------------------------------
As Arun points out, Task isn't a public class, so methods setting/getting this have been moved to JobConf.
was (Author: chris.douglas):
As Arun pointed out, Task isn't a public class, so methods setting/getting this have been moved to JobConf.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606366#action_12606366 ]
Hudson commented on HADOOP-3586:
--------------------------------
Integrated in Hadoop-trunk #523 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/523/])
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606149#action_12606149 ]
Arun C Murthy commented on HADOOP-3586:
---------------------------------------
+1
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Attachment: 3586-1.patch
No tests are included, as this behavior will change in future releases. The javac warnings are from the deprecated methods added in support of the workaround. Fixed the findbugs warning.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Attachment: 3586-0.patch
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Attachment: (was: 3586-0.patch)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-3586:
-------------------------------------
Assignee: Chris Douglas
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Fix For: 0.18.0
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Status: Open (was: Patch Available)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605838#action_12605838 ]
Owen O'Malley commented on HADOOP-3586:
---------------------------------------
Sorry, my comment is out of context. Arkady had asked on the core-dev list:
{quote}
For my education, can someone tell:
was "combiner guaranteed to run once and only once for each map" a documented part of the contract ?
{quote}
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Attachment: 3586-2.patch
As Arun pointed out, Task isn't a public class, so methods setting/getting this have been moved to JobConf.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Status: Patch Available (was: Open)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Status: Patch Available (was: Open)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605833#action_12605833 ]
Owen O'Malley commented on HADOOP-3586:
---------------------------------------
No. In fact, the opposite was the intent. The types in Hadoop are:
k1,v1 -> map -> k2,v2
k2,v2 -> combiner -> k2,v2
k2,v2 -> reduce -> k3,v3
precisely because the combiner is supposed to be an optimization that could run arbitrary numbers of times. In fact, HADOOP-363 was asking for multiple level combiners from and to memory and was filed two years ago.
The problem is that Pig was depending on the previous semantics and it would be difficult for them to change it to support 0.18 in a timely manner. This backward compatibility is just a short term work around to allow Pig (and possibly other applications that break the abstraction) to work with Hadoop 0.18.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605858#action_12605858 ]
Hadoop QA commented on HADOOP-3586:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12384172/3586-0.patch
against trunk revision 668867.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 444 javac compiler warnings (more than the trunk's current 441 warnings).
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2678/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2678/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2678/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2678/console
This message is automatically generated.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606016#action_12606016 ]
Hadoop QA commented on HADOOP-3586:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12384183/3586-1.patch
against trunk revision 669088.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 444 javac compiler warnings (more than the trunk's current 441 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2683/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2683/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2683/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2683/console
This message is automatically generated.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Priority: Blocker (was: Major)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Hadoop Flags: [Reviewed]
Does this qualify as an incompatible change?
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Status: Open (was: Patch Available)
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch, 3586-1.patch, 3586-2.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3586) keep combiner backward compatible
with earlier versions of hadoop
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3586:
----------------------------------
Attachment: 3586-0.patch
This patch adds a property that will restore the previous behavior, which runs the combiner once for each partition in each spill. It will also run the combiner for single records whose serialized size exceeds the buffer size.
It's marked as deprecated, to be removed in a future release.
> keep combiner backward compatible with earlier versions of hadoop
> -----------------------------------------------------------------
>
> Key: HADOOP-3586
> URL: https://issues.apache.org/jira/browse/HADOOP-3586
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: 3586-0.patch
>
>
> In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.
> This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.
> We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.