[kudu-CR] WIP disk failure: add persistent disk states

["Andrew Wong (Code Review)" <ge...@cloudera.org>]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/7270

to look at the new patch set (#2).

Change subject: WIP disk failure: add persistent disk states
......................................................................

WIP disk failure: add persistent disk states

This patch adds a new version of the path set so failed disks are not
used the next time the server is run. This is needed, as data on a
failed disk may be partially written and thus should not be used.

To accomplish this, disk states, path names, and a timestamp are added
to the path set. Additionally, if any disks are failed when loading
instances from disk, an 'invalid' instance with no path set metadata
will be created instead of failing the startup process.

CheckIntegrity() is stripped down to accomodate this. Rather than
comparing all instances against an agreed-upon set of UUIDs, the single
path set with the largest timestamp is used as the main one to compare
against (if only old path sets are available, the first valid one is
used). If no instances are valid, CheckIntegrity() will fail, as all
disks are failed.

The main path set is checked to ensure its integrity (e.g. no
duplicates), after which it is upgraded with the extra metadata if
needed. It is then checked to ensure it is consistent with the valid
instances in terms of having the right UUIDs and paths.

Testing is done in three ways:
- unit testing for the new iteration of CheckIntegrity()
- testing in ts_disk_failure-test to ensure failed disks will not fail
  when starting up the server
- testing in disk_failure-itest to ensure that starting up with a failed
  disk will cause the tablets on the failed disk to replicate

Some notes:
- In the case of a server restart where all healthy disks fail to start
  up and some known failed disks start working again, the server will
  successfully start up with the bad disks (may have partially-written
  data).
- If there are any invalid instances when upgrading the path sets (i.e.
  adding disk states, paths, timestamp), a complete mapping of UUIDs to
  paths will not be achievable, and CheckIntegrity() will fail with an
  IOError.
- The main path set's disk states are updated to reflect the validity of
  the instances. Since data on a failed disk cannot be trusted, disks
  that are successfully read from but are already marked FAILED by the
  main path set are not marked HEALTHY.

WIP because I'd like to add more tests (will add more when other disk
failure patches stabilize). The contents are reviewable and I'd
appreciate feedback on the new integrity checks.

Change-Id: Ifddf0817fe1a82044077f5544c400c88de20769f
---
M src/kudu/fs/block_manager_util-test.cc
M src/kudu/fs/block_manager_util.cc
M src/kudu/fs/block_manager_util.h
M src/kudu/fs/data_dirs-test.cc
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/data_dirs.h
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/fs.proto
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/integration-tests/disk_failure-itest.cc
M src/kudu/tserver/ts_disk_failure-test.cc
12 files changed, 592 insertions(+), 124 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/7270/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifddf0817fe1a82044077f5544c400c88de20769f
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins