[jira] [Commented] (MAPREDUCE-2786) TestDFSIO should also test compression reading/writing from command-line.

["Konstantin Shvachko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436967#comment-13436967 ] 

Konstantin Shvachko commented on MAPREDUCE-2786:
------------------------------------------------

That is a good thing to have an opportunity to benchmark with compression.
Couple suggestions.
# Move all compression configuration logic, including reflections and the cc variable all the way to {{IOMapperBase.configure()}}. Otherwise all this small actions will be counted as execution time.
# You should not work separately with compressed and non-compressed streams inside doIO(). Same {{out}} or {{in}} variables should just point to compressed or not compressed streams. Nesting streams is a regular practice.
# {{getCompression()}} is not used anywhere, should be removed.
# You use {{test.compression}} to get the codec class and {{test.io.compression.class}} to set it. How is going to work? You should make two constants with the property and the default value and use them.
# AppendMapper is not covered. It should be the same as others. Moving the conig logic into {{IOMapperBase}} should make it easy.
                
> TestDFSIO should also test compression reading/writing from command-line.
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2786
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2786
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: benchmarks
>    Affects Versions: 2.0.0-alpha
>            Reporter: Plamen Jeliazkov
>            Priority: Minor
>              Labels: newbie
>             Fix For: 2.1.0-alpha
>
>         Attachments: MAPREDUCE-2786.patch
>
>   Original Estimate: 36h
>  Remaining Estimate: 36h
>
> I thought it might be beneficial to simply alter the code of TestDFSIO to accept any compression codec class and allow testing for compression by a command line argument instead of having to change the config file everytime. Something like "-compression" would do.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira