Encryption at rest

[Jae Joo <ja...@gmail.com>]

The way to have an encrypted index is using OS level encryption.  Is
there any other way to encrypt collection based?

Jae

Re: Encryption at rest

[Jan Høydahl <ja...@cominvent.com>]

No, several attempts but they all come to a halt when realizing that they provide false sense of security due to various flaws, or too big compromise with features and performance. So best practice is OS level.

Jan Høydahl

> 10. sep. 2021 kl. 19:47 skrev Jae Joo <ja...@gmail.com>:
> 
> The way to have an encrypted index is using OS level encryption.  Is
> there any other way to encrypt collection based?
> 
> Jae

Re: Encryption at rest

[David Smiley <ds...@apache.org>]

There isn't quite yet an open-source offering.  We've been doing this at
Salesforce for years with a custom implementation.  We've started anew with
a Directory based abstraction LUCENE-9379[1] (by my colleague Bruno).  A
Google Search shows that DataStax has implemented something similar[1].
Sadly there are some objections to any such feature being contributed to
Lucene/Solr but nonetheless it could be released independently.

[1] https://issues.apache.org/jira/browse/LUCENE-9379
[2] https://docs.datastax.com/en/security/6.7/security/secEncryptSearch.html

It occurred to me recently that users might be able to get this feature
today via Solr's HDFS support[3].  This would require that the HDFS path be
defined in solrconfig.xml (not a global system property) and furthermore
contain the collection name in the path as its own directory.  I bet that
could be templatized, and thus you could have one common solrconfig.xml
file (one so-called "configSet").  With Solr and HDFS, there is no disk
use; it's all on HDFS and cached in memory as well.  HDFS supports
encrypted directories[4].  I'm not sure about key management from the Solr
side of things.

[3]
https://nightlies.apache.org/solr/draft-guides/solr-reference-guide-main/solr-on-hdfs.html
[4]
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/configuring-hdfs-encryption/content/create_an_encryption_zone.html

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Fri, Sep 10, 2021 at 1:54 PM Jae Joo <ja...@gmail.com> wrote:

> The way to have an encrypted index is using OS level encryption.  Is
> there any other way to encrypt collection based?
>
> Jae
>