You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by to...@apache.org on 2016/02/02 07:55:30 UTC
[7/9] incubator-kudu git commit: Enable compression and smaller block
size for composite key index
Enable compression and smaller block size for composite key index
After running the time series workload on d2106 for a couple months,
I found a couple interesting things:
- The composite key index (aka "ad hoc index") was taking 6.4 bytes
per row (vs 5.23 *bits* for the actual data). Compressing it with
'lzop' on that dataset gained a 6.2x savings.
Thus, this patch changes this index to be compressed using LZ4 by default,
which should save space. On the tpch lineitem table, it saved about 15%.
The performance cost should be fairly minimal -- we always random-access
the index blocks, and in the case of a cache miss, the cost of decompression
is tiny compared to the cost of the resulting disk seek.
- Once we reached ~12B rows, the system degenerated into a seeky mess.
Looking at tracing revealed that we spent a lot of time reading
composite indexes, indicating they weren't fitting well in the cache.
I theorize that making these index blocks smaller should decrease the
amount of excess data that gets pulled into the cache when we read them.
Given that these blocks are always random-accessed and never scanned,
using small block sizes makes intuitive sense.
Eventually, both of these options should be table properties, but it
was easier to just set better defaults for now as a quick improvement.
Change-Id: I2b7bfc7a4961c764d262524292ec56e3969af728
Reviewed-on: http://gerrit.cloudera.org:8080/953
Reviewed-by: Jean-Daniel Cryans
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/197b5fa1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/197b5fa1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/197b5fa1
Branch: refs/heads/master
Commit: 197b5fa1a754510a01ac934e8ed5b966de8da29f
Parents: 64bd51c
Author: Todd Lipcon <to...@cloudera.com>
Authored: Tue Aug 11 18:32:07 2015 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Feb 2 02:55:13 2016 +0000
----------------------------------------------------------------------
src/kudu/tablet/diskrowset.cc | 6 ++++++
1 file changed, 6 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/197b5fa1/src/kudu/tablet/diskrowset.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/diskrowset.cc b/src/kudu/tablet/diskrowset.cc
index d397e82..55decd8 100644
--- a/src/kudu/tablet/diskrowset.cc
+++ b/src/kudu/tablet/diskrowset.cc
@@ -50,6 +50,10 @@ DEFINE_double(tablet_delta_store_major_compact_min_ratio, 0.1f,
"can run (Advanced option)");
TAG_FLAG(tablet_delta_store_major_compact_min_ratio, experimental);
+DEFINE_int32(default_composite_key_index_block_size_bytes, 4096,
+ "Block size used for composite key indexes.");
+TAG_FLAG(default_composite_key_index_block_size_bytes, experimental);
+
namespace kudu {
namespace tablet {
@@ -128,6 +132,8 @@ Status DiskRowSetWriter::InitAdHocIndexWriter() {
opts.write_posidx = false;
opts.storage_attributes.encoding = PREFIX_ENCODING;
+ opts.storage_attributes.compression = LZ4;
+ opts.storage_attributes.cfile_block_size = FLAGS_default_composite_key_index_block_size_bytes;
// Create the CFile writer for the ad-hoc index.
ad_hoc_index_writer_.reset(new cfile::CFileWriter(