[kudu-CR] [encryption] KUDU-3331 Encrypt file system

["Attila Bukor (Code Review)" <ge...@cloudera.org>]

Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Bankim Bhavsar, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17974

to look at the new patch set (#4).

Change subject: [encryption] KUDU-3331 Encrypt file system
......................................................................

[encryption] KUDU-3331 Encrypt file system

de02a34 introduced encryption support to Env in a self-contained way,
but it's not used across Kudu.

This commit integrates this encryption support into the project and
modifies several test suites to also run tests with encryption enabled.

I also changed the way the "encrypted" flag in *FileOptions works:
previously, the default was false, as files were not encrypted by
default. As this commit introduces the "--encrypt_data_at_rest" flag,
which is false by default, the encrypted flag now means that a file is
encrypted or not if data at rest encryption is enabled. The default is
now true, which results in less code change and not having to complicate
FileCache.

When encryption is enabled, the following files are encrypted:

- WAL and blocks as they contain user data, which is considered
  sensitive.
- LBM block metadata as FileCache is used to access them, so by
  encrypting these files, we can keep FileCache simpler as it doesn't
  have to be able to handle both encrypted and non-encrypted files.

Instance and block manager instance files don't need to be encrypted,
and neither do consensus and tablet metadata files.

As most PBC files are non-encrypted, PBUtil disables encryption by
default. This leads to an edge-case in the PBC CLI tool, which can dump
and edit PBC files. This needs to be able to recognize whether the file
is encrypted. As encryption headers are not yet implemented, I added a
hack which checks the file name and treats the file as encrypted if it
ends with ".metadata" (LBM block metadata extension) and non-encrypted
otherwise.

Change-Id: I909d0c4af0c1fca0d14c99a6627842dbe2ed7524
---
M src/kudu/consensus/consensus_meta-test.cc
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/integration-tests/security-itest.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/ranger_client.cc
M src/kudu/security/test/mini_kdc.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_pbc.cc
M src/kudu/util/env-test.cc
M src/kudu/util/env.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M src/kudu/util/file_cache-test.cc
M src/kudu/util/pb_util-test.cc
M src/kudu/util/pb_util.cc
M src/kudu/util/rolling_log.cc
M src/kudu/util/yamlreader-test.cc
20 files changed, 282 insertions(+), 103 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/74/17974/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I909d0c4af0c1fca0d14c99a6627842dbe2ed7524
Gerrit-Change-Number: 17974
Gerrit-PatchSet: 4
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)