You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Subir S (Created) (JIRA)" <ji...@apache.org> on 2012/02/25 17:01:49 UTC

[jira] [Created] (MAPREDUCE-3924) Partioning Failing in certain scenarios for Hadoop Streaming using Ruby

Partioning Failing in certain scenarios for Hadoop Streaming using Ruby
-----------------------------------------------------------------------

                 Key: MAPREDUCE-3924
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3924
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.2
         Environment: Linux OS
44 Node cluster
Hadoop version 0.20.3-cdhu3 is the version
            Reporter: Subir S


We noticed a wierd scenario. The partitioning option fails in scenario below. The map reduce just aggregates values like a word count. However the key is having 2 parts.

Eg:
Input Records: (<key1|key2><tab><qty1|amt1|qty2|amt2|qty3|amt3>
1201|420 1|24.0|2|26.0|0|0.0
1200|420 1|24.0|2|26.0|0|0.0
1200|420 1|25.0|3|52.0|1|55.0
1403|400 1|25.0|3|52.0|1|55.0
1201|420 1|24.0|2|26.0|0|0.0
1201|420 1|24.0|2|26.0|0|0.0
1403|400 1|25.0|3|52.0|1|55.0

Partioning option: -k1,1 and Comparator option -k1,1n -k2,2n works fine and the output we get is expected output
1200|420 2|49.0|5|78.0|1|55.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0

However if we use
Partioning option: -k1,1 -k2,2 and Comparator option -k1,1n -k2,2n then the output is faulty. Seems the same key does not go to same reducer in output we see duplicate records, however this does not happen to all records. A hypothetical faulty output based on above input is:
1200|420 1|24.0|2|26.0|0|0.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0
1200|420 1|25.0|3|52.0|1|55.0

See that records with the key (1200|420) is not aggregated. This can happen only if the record does not go to the same reducer after partioning.

Any clue why this is happening? I could not understanding what is going wrong.






--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-3924) Partioning Failing in certain scenarios for Hadoop Streaming using Ruby

Posted by "Subir S (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Subir S resolved MAPREDUCE-3924.
--------------------------------

    Resolution: Not A Problem

My mistake. This is not a bug. It was related to how we specified the key field separaters which invoking the streaming. We had overridden that to be a '|' instead of default 'tab' and that caused all confusion.
                
> Partioning Failing in certain scenarios for Hadoop Streaming using Ruby
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3924
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3924
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>         Environment: Linux OS
> 44 Node cluster
> Hadoop 0.20.2-cdh3u0 is the version
>            Reporter: Subir S
>              Labels: hadoop, map, reduce, ruby, streaming
>
> We noticed a wierd scenario. The partitioning option fails in scenario below. The map reduce just aggregates values like a word count. However the key is having 2 parts.
> Eg:
> Input Records: (<key1|key2><tab><qty1|amt1|qty2|amt2|qty3|amt3>
> 1201|420 1|24.0|2|26.0|0|0.0
> 1200|420 1|24.0|2|26.0|0|0.0
> 1200|420 1|25.0|3|52.0|1|55.0
> 1403|400 1|25.0|3|52.0|1|55.0
> 1201|420 1|24.0|2|26.0|0|0.0
> 1201|420 1|24.0|2|26.0|0|0.0
> 1403|400 1|25.0|3|52.0|1|55.0
> Partioning option: -k1,1 and Comparator option -k1,1n -k2,2n works fine and the output we get is expected output
> 1200|420 2|49.0|5|78.0|1|55.0
> 1201|420 3|72.0|6|78.0|0|0.0
> 1403|400 2|50.0|6|104.0|2|110.0
> However if we use
> Partioning option: -k1,1 -k2,2 and Comparator option -k1,1n -k2,2n then the output is faulty. Seems the same key does not go to same reducer in output we see duplicate records, however this does not happen to all records. A hypothetical faulty output based on above input is:
> 1200|420 1|24.0|2|26.0|0|0.0
> 1201|420 3|72.0|6|78.0|0|0.0
> 1403|400 2|50.0|6|104.0|2|110.0
> 1200|420 1|25.0|3|52.0|1|55.0
> See that records with the key (1200|420) is not aggregated. This can happen only if the record does not go to the same reducer after partioning.
> Any clue why this is happening? I could not understanding what is going wrong.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira