You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@chukwa.apache.org by "Jie Huang (JIRA)" <ji...@apache.org> on 2012/07/16 08:58:33 UTC

[jira] [Updated] (CHUKWA-647) Spread out intermediate data with the same ReduceType into different Reduce Tasks

     [ https://issues.apache.org/jira/browse/CHUKWA-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jie Huang updated CHUKWA-647:
-----------------------------

    Attachment: Chukwa-647.patch

Here attaches a simple workaround. If the key contains the specific mark, the partitioner will include the key as well. OR Another option is to include part of the key content.Any other idea?
                
> Spread out intermediate data with the same ReduceType into different Reduce Tasks
> ---------------------------------------------------------------------------------
>
>                 Key: CHUKWA-647
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-647
>             Project: Chukwa
>          Issue Type: Improvement
>          Components: Data Processors
>    Affects Versions: 0.4.0, 0.6.0
>            Reporter: Jie Huang
>            Priority: Minor
>         Attachments: Chukwa-647.patch
>
>
> We have found that if we partitioned the map output data according to ReduceType, we can see the data skew in some HiTune cases. Then one or two Reduce Tasks slow down the whole Demux job somehow, since those reduce tasks have to process more input-data.    

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira