[jira] Commented: (PIG-467) compressing intermediate results

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635619#action_12635619 ] 

Arun C Murthy commented on PIG-467:
-----------------------------------

It definitely is better to use lzo rather than zlib for compressing intermediate map-outputs. However, lzo might not be available on the cluster, and the right 32/64 bit lzo libraries need to installed. Hence, it would be pertinent to default to zlib but have an easy way for the user or admin to change it to lzo.

> compressing intermediate results
> --------------------------------
>
>                 Key: PIG-467
>                 URL: https://issues.apache.org/jira/browse/PIG-467
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>             Fix For: types_branch
>
>
> It is recommended with Hadoop 18 and later versions to compress data passed between Map and Reduce. We need to test to make sure that it gives performance gain

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.