You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by tm...@apache.org on 2018/10/17 17:02:58 UTC
[03/11] impala git commit: IMPALA-7708: Switch to faster deflater
compression level for incr stats
IMPALA-7708: Switch to faster deflater compression level for incr stats
On a table with 3000 partitions and ~150 columns, we noticed that the
BEST_SPEED deflater strategy is ~8x faster with ~4% compression ratio
penalty. Given these results, this patch switches the default to
BEST_SPEED from BEST_COMPRESSION.
Change-Id: Ife688aca3aed0e1e8af26c8348b850175d84b4ad
Reviewed-on: http://gerrit.cloudera.org:8080/11685
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/3b6c0f62
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/3b6c0f62
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/3b6c0f62
Branch: refs/heads/master
Commit: 3b6c0f6296e807b25f3e40bd614b2571f4f01d48
Parents: 31dfa3e
Author: Bharath Vissapragada <bh...@cloudera.com>
Authored: Mon Oct 15 10:47:19 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 01:13:16 2018 +0000
----------------------------------------------------------------------
fe/src/main/java/org/apache/impala/util/CompressionUtil.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/3b6c0f62/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
index 86d8477..f6e11a1 100644
--- a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
@@ -40,9 +40,11 @@ public class CompressionUtil {
public static byte[] deflateCompress(byte[] input) {
if (input == null) return null;
ByteArrayOutputStream bos = new ByteArrayOutputStream(input.length);
- // TODO: Benchmark other compression levels.
+ // Experiments on a wide partitioned table with incremental stats showed that the
+ // Deflater with 'BEST_SPEED' level provided reasonable compression ratios at much
+ // faster speeds compared to other modes like BEST_COMPRESSION/DEFAULT_COMPRESSION.
DeflaterOutputStream stream =
- new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_COMPRESSION));
+ new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED));
try {
stream.write(input);
stream.close();