You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Dong Xiang (JIRA)" <ji...@apache.org> on 2012/09/14 08:45:08 UTC

[jira] [Commented] (HADOOP-8800) Dynamic Compress Stream

    [ https://issues.apache.org/jira/browse/HADOOP-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455609#comment-13455609 ] 

Dong Xiang commented on HADOOP-8800:
------------------------------------

Looking forward to hearing more details about this proposed feature. 
                
> Dynamic Compress Stream
> -----------------------
>
>                 Key: HADOOP-8800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8800
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 2.0.1-alpha
>            Reporter: yankay
>              Labels: patch
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> We use compress in MapReduce in some case because It use CPU to improve IO throughput.
> But we can only set one compress algorithm in configure file. The hadoop cluster is changing every time.  So a compress algorithm may not work well in all case. 
> Why not provide an algorithm named dynamic. It can change compress level and algorithm dynamicly based on performance. Like tcp, it starts up slowly, and try run faster and faster. It can make the io faster by choose a more suitable compress algorithm.
> I would write a detail design here, and try to submit a patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira