Fw: High level interface to Parquet encryption

[Gidon Gershinsky <gg...@gmail.com>]

Hi all,

Now that the low level interface to Parquet encryption is merged in
parquet-cpp, and close to completion in parquet-mr, we need to get back to
the subject of a high level interface, that allows to use Parquet
encryption in a simple, almost transparent way; and helps with management
of encryption keys.

What has changed in this field since June'19, when we have last
discussed it?
- the basic Parquet encryption layer and its low level interface are mostly
complete
- the two alternatives to high level interfaces we had (properties-driven,
and schema-driven), are not mutually exclusive anymore. Together with
Xinli, Gabor and Maya, we have managed to create a simple Crypto Factory
interface mechanism (PARQUET-1817
<https://issues.apache.org/jira/browse/PARQUET-1817>, already merged in
parquet-mr/encryption), that allows to plug in any of the two alternatives
- or any other implementation of a high level encryption interface.
- the properties-driven interface, and the key management tools used for
its implementation, have matured significantly, and are already deployed in
production.
- I presume the schema-driven interface (crypto-interface with schema
activation) has significantly matured as well.

The draft design of the Properties-driven encryption is here:
https://docs.google.com/document/d/1boH6HPkG0ZhgxcaRkGk3QpZ8X_J91uXZwVGwYN45St4/edit?usp=sharing

   - Key management tools (leveraged to build the properties-driven
   encryption, but have a wider applicability), design:

   https://docs.google.com/document/d/1bEu903840yb95k9q2X-BlsYKuXoygE4VnMDl9xz_zhk/edit?usp=sharing

   - Code: the draft pull request that implements Properties-driven
   encryption  (and Key management tools) is here:
   https://github.com/apache/parquet-mr/pull/615


Xinli informs that the Schema-driven design doc is ready too, and a link
will be sent soon.


All feedback from the community will be appreciated.

Cheers, Gidon.

Re: Fw: High level interface to Parquet encryption

[Xinli shang <sh...@uber.com.INVALID>]

Thanks Gidon for sending out! Here is the link for the Schema-driven
design doc
<https://docs.google.com/document/d/17GTQAezl1ZC1pMNHjYU_bPVxMU6DIPjtXOiLclXUlyA>
.



On Mon, May 11, 2020 at 6:14 AM Gidon Gershinsky <gg...@gmail.com> wrote:

> Hi all,
>
> Now that the low level interface to Parquet encryption is merged in
> parquet-cpp, and close to completion in parquet-mr, we need to get back to
> the subject of a high level interface, that allows to use Parquet
> encryption in a simple, almost transparent way; and helps with management
> of encryption keys.
>
> What has changed in this field since June'19, when we have last
> discussed it?
> - the basic Parquet encryption layer and its low level interface are mostly
> complete
> - the two alternatives to high level interfaces we had (properties-driven,
> and schema-driven), are not mutually exclusive anymore. Together with
> Xinli, Gabor and Maya, we have managed to create a simple Crypto Factory
> interface mechanism (PARQUET-1817
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_PARQUET-2D1817&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=HgtpTTM_Ioa61sMBfZKozWM5tNOs8ZanDhyb9mwcOtA&e=
> >, already merged in
> parquet-mr/encryption), that allows to plug in any of the two alternatives
> - or any other implementation of a high level encryption interface.
> - the properties-driven interface, and the key management tools used for
> its implementation, have matured significantly, and are already deployed in
> production.
> - I presume the schema-driven interface (crypto-interface with schema
> activation) has significantly matured as well.
>
> The draft design of the Properties-driven encryption is here:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1boH6HPkG0ZhgxcaRkGk3QpZ8X-5FJ91uXZwVGwYN45St4_edit-3Fusp-3Dsharing&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=ZRk5BcX4fZtHBdesaR1Mb1-0k0FAgrIYas6dGwazcqE&e=
>
>    - Key management tools (leveraged to build the properties-driven
>    encryption, but have a wider applicability), design:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1bEu903840yb95k9q2X-2DBlsYKuXoygE4VnMDl9xz-5Fzhk_edit-3Fusp-3Dsharing&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=dOqanWhKkhBmprgIE_qO1rbEq08dAB60PyFJh3dvUYo&e=
>
>    - Code: the draft pull request that implements Properties-driven
>    encryption  (and Key management tools) is here:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_pull_615&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=CljrTqJHk--geQJLa18DN6iKuPcKH-_J2Lsneowqmck&e=
>
>
> Xinli informs that the Schema-driven design doc is ready too, and a link
> will be sent soon.
>
>
> All feedback from the community will be appreciated.
>
> Cheers, Gidon.
>


-- 
Xinli Shang