[jira] Commented: (CHUKWA-146) All hadoop logs should use a different RecordType

["Eric Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CHUKWA-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699942#action_12699942 ] 

Eric Yang commented on CHUKWA-146:
----------------------------------

+1 on the design.

> All hadoop logs should use a different RecordType
> -------------------------------------------------
>
>                 Key: CHUKWA-146
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-146
>             Project: Hadoop Chukwa
>          Issue Type: Improvement
>            Reporter: Jerome Boulon
>            Priority: Critical
>
> All hadoop logs are using the same RecordType, so only 1 Reducer is used to process all log files (other than DN,NN,Audit).
> This cause a SKU issue at the M/R level.
> So all hadoop logs should use a different RecordType.
> Note: 
> - using the cluster information in the ChukwaRecordPartitioner will also help.
> - using a predefine list of recordType/reducer association will also help by avoiding to have 2 log RecordType going to the same reducer,
> the dynamic affectation ( ( hashCode() & Integer.MAX_VALUE) % numReduceTasks) could be used at a fallback mechanism

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.