[jira] [Commented] (LUCENE-9379) Directory based approach for index encryption

["Bruno Roustant (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136783#comment-17136783 ] 

Bruno Roustant commented on LUCENE-9379:
----------------------------------------

So I plan to implement an EncryptingDirectory extending FilterDirectory.

 

+Encryption method:+

AES CTR (counter)
 * This mode is approved by NIST. ([https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29])
 * AES encryption has the same size as the original clear text (though the last block is padded to 128 bits). So we can use the same file pointers.
 * CTR mode allows random access to encrypted blocks (128 bits blocks).
 * IV (initialisation vector) must be random, and is stored at the beginning of the encrypted file because it can be public.
 * It is appropriate to encrypt streams.

 

+API:+ 

I don’t anticipate any API change.

 

+How to provide encryption keys:+

EncryptingDirectory would require a delegate Directory, an encryption key supplier, and a Cipher pool (for performance).

For the callers to pass the encryption keys, I see two ways:

1- In Solr, declare a DirectoryFactory in solrconfig.xml that creates EncryptingDirectory. This factory is able to determine the encryption key per file based on the path. It is the responsibility of this factory to access the keys (e.g. stored in safe DB, received with an admin handler, read from properties, etc). The Cipher pool is hold by the DirectoryFactory.

2- More generally the EncryptingDirectory can be created to wrap a Directory when opening a segment (e.g. in PostingsFormat/DocValuesFormat fieldsConsumer()/fieldsProducer(), in StoredFieldFormat fieldsReader()/fieldsWriter(), etc). In this case the PostingsFormat/DocValuesFormat/StoredFieldFormat extension determines the encryption key based on the SegmentInfo. A custom Codec can be created to handle encrypting formats. The Cipher pool is hold either in the Codec or in the Format.

 

+Code:+

I will inspire from Apache commons-crypto CtrCryptoOutputStream, although not directly using it because it is an OutputStream while we need an IndexOutput. And we can probably simplify since we have a specific use-case compared to this lib wide usage.

> Directory based approach for index encryption
> ---------------------------------------------
>
>                 Key: LUCENE-9379
>                 URL: https://issues.apache.org/jira/browse/LUCENE-9379
>             Project: Lucene - Core
>          Issue Type: New Feature
>            Reporter: Bruno Roustant
>            Assignee: Bruno Roustant
>            Priority: Major
>
> The goal is to provide optional encryption of the index, with a scope limited to an encryptable Lucene Directory wrapper.
> Encryption is at rest on disk, not in memory.
> This simple approach should fit any Codec as it would be orthogonal, without modifying APIs as much as possible.
> Use a standard encryption method. Limit perf/memory impact as much as possible.
> Determine how callers provide encryption keys. They must not be stored on disk.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org