You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Richard Ding (JIRA)" <ji...@apache.org> on 2010/08/26 23:54:55 UTC
[jira] Updated: (PIG-1458) aggregate files for replicated join
[ https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1458:
------------------------------
Attachment: PIG-1458.patch
This patch uses the new multi-file-combiner (PIG-1518) to concatenate many small files for replicated join. This is based on the assumption that the total size of the replicated files should be small enough to fit into main memory.
> aggregate files for replicated join
> -----------------------------------
>
> Key: PIG-1458
> URL: https://issues.apache.org/jira/browse/PIG-1458
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1458.patch
>
>
> We have noticed that if the smaller data in replicated join has many files, this puts unneeded burden on the name node. pre-aggregating the files can improve the situation
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.