[jira] Commented: (PIG-1458) aggregate files for replicated join

["Richard Ding (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897451#action_12897451 ] 

Richard Ding commented on PIG-1458:
-----------------------------------

The proposal is to run another map-reduce job to merge the small files before the replicated join. This additional job will be added to the MR plan at the compile time.

We consider three cases of a replicated join: 

# The right input is a map-only job and input files exist at the compile time.
# The right input is a map-only job and input files do not exist at the compile time.
# The right input is a map-reduce job.

For 1., if the number of files exceeds the threshold specified in the property file (_pig.frjoin.merge.files.threshold_), a merge job is added between right input job and FR join job.

For 3., if the number of reducers exceeds the threshold specified in the property file (_pig.frjoin.merge.files.threshold_), a merge job is added between right input job and FR join job.

For 2., if the flag specified in the property file (_pig.frjoin.merge.files.optimistic_) is false,  a merge job is added between right input job and FR join job. The default value of this flag is false. 



> aggregate files for replicated join
> -----------------------------------
>
>                 Key: PIG-1458
>                 URL: https://issues.apache.org/jira/browse/PIG-1458
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Olga Natkovich
>            Assignee: Richard Ding
>             Fix For: 0.8.0
>
>
> We have noticed that if the smaller data in replicated join has many files, this puts  unneeded burden on the name node. pre-aggregating the files can improve the situation

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.