How the key rotation works when using Parquet Modular Encryption

[Priyanshu Sharma <pr...@licious.com.INVALID>]

With Parquet Modular Encryption
1. With each key rotation , Is it possible to avoid encryption and
decryption of existing data?

2. If master key rotation does not require modification of the data file
then how would the KMS work.

3. Do we have any constraints for key structure while updating a key.

It would be better if you could provide a git link having the interface to
implement KMS. I am already following this git page
https://github.com/apache/parquet-format/blob/master/Encryption.md but
still have a few doubts.

Re: How the key rotation works when using Parquet Modular Encryption

[Gidon Gershinsky <gg...@gmail.com>]

On Wed, Nov 29, 2023 at 5:40 PM Priyanshu Sharma
<pr...@licious.com.invalid> wrote:

> With Parquet Modular Encryption
> 1. With each key rotation , Is it possible to avoid encryption and
> decryption of existing data?
>
Yes

>
> 2. If master key rotation does not require modification of the data file
> then how would the KMS work.
>
- Basic key rotation simply means the master key version is updated in the
KMS, so the future parquet files are encrypted with the rotated master key
(namely, their data keys will be encrypted with the new master key version).
- In addition, if your threat model requires to re-wrap data keys of
existing parquet files with the rotated master key - this can be done
without modification of the parquet files if they were encrypted in an
"external key material" mode,
parquet.encryption.key.material.store.internally=false
(see
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/README.md#class-propertiesdrivencryptofactory
),
the data keys (encrypted with master keys in KMS) are stored in separate
small key_material files. The key re-wrapping will re-encrypt the data keys
with the rotated master key, and replace the key_material files.

>
> 3. Do we have any constraints for key structure while updating a key.
>
This is up to the KMS service implementation.

>
> It would be better if you could provide a git link having the interface to
> implement KMS. I am already following this git page
> https://github.com/apache/parquet-format/blob/master/Encryption.md but
> still have a few doubts.
>
The links and basic details can be found in
https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#columnar-encryption



Cheers, Gidon