You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Binglin Chang (Commented) (JIRA)" <ji...@apache.org> on 2011/12/15 06:07:31 UTC
[jira] [Commented] (HADOOP-7657) Add support for LZ4 compression
[ https://issues.apache.org/jira/browse/HADOOP-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169970#comment-13169970 ]
Binglin Chang commented on HADOOP-7657:
---------------------------------------
I have integrated lz4 in native task optimizition(MAPREDUCE-2841), and it did show better results compare to snappy. Here is some test results for terasort input data:
[ RUN ] Perf.Compressions
11/12/15 13:04:24 INFO Generate data time: 2.12431s
11/12/15 13:04:26 INFO memset buffer to prevent missing page time: 2.08189s
11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.SnappyCodec
11/12/15 13:04:26 INFO snappy compress origin/compressed time: 0.16029s size: 100.000M/21.944M speed: 623.86M/136.90M
11/12/15 13:04:26 INFO snappy decompress orig/uncompressed time: 0.05860s size: 21.944M/100.000M speed: 374.49M/1706.57M
11/12/15 13:04:26 INFO ratio: 0.219
11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.Lz4Codec
11/12/15 13:04:27 INFO snappy compress origin/compressed time: 0.13551s size: 100.000M/18.952M speed: 737.96M/139.86M
11/12/15 13:04:27 INFO snappy decompress orig/uncompressed time: 0.03126s size: 18.952M/100.000M speed: 606.24M/3198.87M
11/12/15 13:04:27 INFO ratio: 0.190
[ OK ] Perf.Compressions (4665 ms)
[----------] 1 test from Perf (4665 ms total)
> Add support for LZ4 compression
> -------------------------------
>
> Key: HADOOP-7657
> URL: https://issues.apache.org/jira/browse/HADOOP-7657
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Mr Bsd
> Labels: compression
>
> According to several benchmark sites, LZ4 seems to overtake other fast compression algorithms, especially in the decompression speed area. The interface is also trivial to integrate (http://code.google.com/p/lz4/source/browse/trunk/lz4.h) and there is no license issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira