[kudu-CR] WIP KUDU-3371 [fs] Use RocksDB to store LBM metadata

["Yingchun Lai (Code Review)" <ge...@cloudera.org>]

Hello Alexey Serbin, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18569

to look at the new patch set (#30).

Change subject: WIP KUDU-3371 [fs] Use RocksDB to store LBM metadata
......................................................................

WIP KUDU-3371 [fs] Use RocksDB to store LBM metadata

Since LogBlockContainer store block records sequentially in
metadata file, the live blocks maybe in a very low ratio,
and it cause disk space wasting and long time bootstrap.

This patch use RocksDB to store LBM metadata, a new
item will be Put() into RocksDB when a new block created
in LBM, and the item will be Delete()d from RocksDB when
the block removed from LBM. Data in RocksDB can be
maintained in RocksDB itself, i.e. deleted items will be
GCed so doesn't need rewriting as how we do it in current
LBM.

The implemention also reuse most logic of LBM, the main
difference is store Block records metadata in RocksDB.
Introduce a new class LogrBlockManager that stores
metadata in RocksDB, the main idea:
a. Create container
   Data file is created as before, metadata is stored
   in keys prefixed by the container's id, append the
   block id, e.g. <container_id>.<block_id>. Make sure
   there is no such keys in RocksDB before this
   container created.
b. Open container
   Make sure the data file is healthy.
c. Deconstruct container
   If the container is dead (full and no live blocks),
   remove the data file, and clean up keys prefixed by
   the container's id.
d. Load container (by ProcessRecords())
   Iterate the RocksDB in the key range
   [<container_id>, <next_container_id>), only live
   block records will be populated, we can use them
   as before.
e. Create blocks in a container
   Put() serialized BlockRecordPB records into RocksDB
   in batch, keys are in form of
   '<container_id>.<block_id>' as mentioned above.
f. Remove blocks from a container
   Contruct the keys by container's id and block's
   id, Delete() them from RocksDB in batch.

This patch contains the following changes:
- Adds a new block manager type named 'logr', it use RocksDB
  to store LBM metadata, it is also specified by flag
  '--block_manager'.
- block_manager-test supports to test LogrBlockManager
- block_manager-stress-test supports to test LogrBlockManager
- log_block_manager-test supports to test LogrBlockManager
- tablet_server-test supports to test LogrBlockManager
- dense_node-itest supports to test LogrBlockManager
- kudu-tool-test supports to test LogrBlockManager

It's optional to use RocksDB, we can use the former LBM as
before, we can introduce more tools to convert data between
the two implemention in the future.

The optimization is obvious as shown in JIRA KUDU-3371, it
shows that reopen staged reduced upto 90% time cost.

Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
---
M src/kudu/benchmarks/CMakeLists.txt
M src/kudu/client/CMakeLists.txt
M src/kudu/consensus/CMakeLists.txt
M src/kudu/fs/CMakeLists.txt
M src/kudu/fs/block_manager-stress-test.cc
M src/kudu/fs/block_manager-test.cc
M src/kudu/fs/block_manager.h
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/dir_manager.cc
M src/kudu/fs/dir_manager.h
M src/kudu/fs/dir_util.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_report.cc
M src/kudu/fs/fs_report.h
M src/kudu/fs/log_block_manager-test-util.cc
M src/kudu/fs/log_block_manager-test-util.h
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/dense_node-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/server/CMakeLists.txt
M src/kudu/tablet/compaction-test.cc
M src/kudu/tools/CMakeLists.txt
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_test_util.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/CMakeLists.txt
31 files changed, 1,820 insertions(+), 445 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/18569/30
-- 
To view, visit http://gerrit.cloudera.org:8080/18569
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
Gerrit-Change-Number: 18569
Gerrit-PatchSet: 30
Gerrit-Owner: Yingchun Lai <la...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>