[jira] [Updated] (ZOOKEEPER-1688) Transparent encryption of on-disk files

["Andrew Purtell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated ZOOKEEPER-1688:
--------------------------------------

    Attachment: ZOOKEEPER-1688.patch

Attaching a patch that is suitable for review. The new unit tests pass and it has been lightly tested. I expect to start full cluster testing of different failure scenarios next week.

The changes are straightforward and follow the description above. The approach to extending the file persistence classes has been reworked a few times to minimize the number of changes needed. We allow for wrapper streams to be inserted just after FileInputStreams are created for reading log and snapshot files.

The most significant detail is how preallocation and encryption play together - or, to say, they do not. If preallocation is allowed, at read, when a CipherInputStream is filtering the input, FileTxnLog will read past the last transaction into the padding and the padding will be transformed by the cipher into garbage. A solution to this problem requires more substantial changes in core ZooKeeper code that I wanted to allow (e.g. an end-of-log record). As a consequence, if the encrypted persistence option is selected preallocation won't happen. To mitigate the resulting substantial increase in transaction latency, we introduce a new system property that specifies a delay interval between a commit to the log and fsync, defaulting to 100ms, that is only active and honored if the encrypting persistence option is selected. For any commit, the delay before fsync will be at most this interval. Setting the interval to 0 fsyncs immediately upon every commit.
                
> Transparent encryption of on-disk files
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1688
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1688
>             Project: ZooKeeper
>          Issue Type: New Feature
>    Affects Versions: 3.5.0
>            Reporter: Andrew Purtell
>         Attachments: ZOOKEEPER-1688.patch
>
>
> We propose to introduce optional transparent encryption of snapshots and commit logs on disk. The goal is to protect against the leakage of sensitive information from files at rest, due to accidental misconfiguration of filesystem permissions, improper decommissioning, or improper disk disposal. This change would introduce a new ServerConfig option that allows the administrator to select the desired persistence implementation by classname, and new persistence classes extending the File* classes that wrap current formats in encrypted containers. Otherwise and by default the current File* classes will be used without change. If enabled, transparent encryption of all on disk structures will be accomplished with a shared cluster key made available to the quorum peers via the Java Keystore (supporting various store options, including hardware security module integration). Small modifications to the LogFormatter and SnapshotFormatter utilities will be needed. A new utility for offline key rotation will also be provided.
> These changes will not introduce any new dependencies. The standard Java Cryptographic Extensions (JCE) are sufficient for implementation and can benefit from potential acceleration options provided by JCE now or future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira