You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ankit Kamboj (JIRA)" <ji...@apache.org> on 2012/07/02 20:48:31 UTC
[jira] [Commented] (MAPREDUCE-4354) Performance improvement with
compressor object reinit restriction
[ https://issues.apache.org/jira/browse/MAPREDUCE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405234#comment-13405234 ]
Ankit Kamboj commented on MAPREDUCE-4354:
-----------------------------------------
This makes sense. The reinit for LZO should only reset buffers instead of recreating them.
So, I have removed the 'if' condition for GZIP in CodecPool and instead changed reinit logic in LZOCompressor.
Please find attached the corresponding patch.
> Performance improvement with compressor object reinit restriction
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4354
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.20.205.0
> Reporter: Ankit Kamboj
> Priority: Minor
> Labels: performance
> Fix For: 0.20.205.0
>
> Attachments: codec_reinit_diff
>
>
> HADOOP-5879 patch aimed at picking the conf (instead of default) settings for GzipCodec. It also involved re-initializing the recycled compressor object.
> On our performance tests, this re-initialization led to performance degradation of 15% for LzoCodec because re-initialization for Lzo involves reallocation of buffers. LzoCodec takes the initial settings from config so it is not necessary to re-initialize it. This patch checks for the codec class and calls reinit only if the codec class is Gzip. This led to significant performance improvement of 15% for LzoCodec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira