You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Xiao Kang (JIRA)" <ji...@apache.org> on 2010/03/29 09:59:27 UTC
[jira] Created: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
hadoop zlib compression does not fully utilize the buffer
---------------------------------------------------------
Key: HADOOP-6662
URL: https://issues.apache.org/jira/browse/HADOOP-6662
Project: Hadoop Common
Issue Type: Improvement
Components: io
Affects Versions: 0.20.2
Reporter: Xiao Kang
org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851334#action_12851334 ]
Hadoop QA commented on HADOOP-6662:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12440187/ZlibCompressor.patch
against trunk revision 927979.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/434/console
This message is automatically generated.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
> Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Kang updated HADOOP-6662:
------------------------------
Attachment: ZlibCompressor.patch
Patch attached.
needsInput() check the uncompressedDirectBuf, if it is full return false, else copy data from saved userBuf and then recheck.
A special case, that the input uncompressedDirectBuf is not all comsumed by zlib due to output buffer is not enough, should be respected. It may be the reason the original code just return false if uncompressedBufLen > 0.
After JNI compress invoked, uncompressedBufLen will be set back to the remaining input data length that not consumed by zlib. So if uncompressedBufLen > 0 after deflateBytesDirect() invoked, a flag keepUncompressedBuf is setted true to indicate no input needed and compress() should be invoked again to compress the remainling input data.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
> Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Kang reopened HADOOP-6662:
-------------------------------
Thanks for Hong Tang. Patch is not attached in HADOOP-4196 and the issue is still unresolved in release-0.20.2.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
> Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851277#action_12851277 ]
Xiao Kang commented on HADOOP-6662:
-----------------------------------
Thank you for your notice! However no patch attached for https://issues.apache.org/jira/browse/HADOOP-4196. HADOOP-4196 discussed more issues, and this issuse foucus on the zlib compression buffer issuse.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Kang updated HADOOP-6662:
------------------------------
Status: Patch Available (was: Reopened)
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
> Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851016#action_12851016 ]
Hong Tang commented on HADOOP-6662:
-----------------------------------
This is duplicate for HADOOP-4196.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Kang updated HADOOP-6662:
------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Duplicated with HADOOP-6683.
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
> Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6662) hadoop zlib compression does not
fully utilize the buffer
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang resolved HADOOP-6662.
-------------------------------
Resolution: Duplicate
> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
> Key: HADOOP-6662
> URL: https://issues.apache.org/jira/browse/HADOOP-6662
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.20.2
> Reporter: Xiao Kang
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer.
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.