You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2019/01/18 00:50:21 UTC

Adding KMS support to generic filesystem interface

Hi,
I'd like to add support for creating files using a cloud Key Management
System.
A KMS allows you to audit, create, rotate, and disable encryption keys.
Both AWS and GCP have such a service..

I wanted to show the community what I've been working on and see if there
are any comments or objection before submitting a PR.
https://github.com/udim/beam/commit/d29f1ef26c58489416a2d413eb029596d96e1f25

Reference docs:
AWS S3:
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
GCP GCS:
https://cloud.google.com/storage/docs/encryption/using-customer-managed-keys#add-object-key

Re: Adding KMS support to generic filesystem interface

Posted by Ismaël Mejía <ie...@gmail.com>.
Hello Udi,

I took a look at your PR and let some coments.Please take a look at
them because I think you can achieve what you want in a simpler way
and without radical changes to the FileSystem (and File-related)
interface(s).

Let's continue the discussion in the PR and then we bring back the
subject to the mailing list.

Regards,
Ismaël



On Fri, Jan 18, 2019 at 7:06 PM Udi Meiri <eh...@google.com> wrote:
>
> Hi Ismaël,
> I'd like your feedback, especially from the AWS perspective.
> I wasn't aware of BEAM-3821, but I did create a JIRA for Cloud KMS support on GCS: https://issues.apache.org/jira/browse/BEAM-5959
>
> Some details of my plan for KMS support:
> 1. Add KMS settings to sources and sinks.
> 2. Add a --kmsKey flag that is passed to the runner and applies to pipeline state.
>
> On Fri, Jan 18, 2019 at 8:24 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>> Hello Udi,
>>
>> I implemented the support for KMS in Amazon and I am really interested
>> in check your PR. However I won't have time to do it until next
>> monday. I hope waiting a bit is ok with you if you want some feedback
>> from me.
>>
>> I am curious if you considered or are aware of this issue:
>> BEAM-3821 Support a pluggable key management system (KMS)
>> https://issues.apache.org/jira/browse/BEAM-3821
>>
>>
>> On Fri, Jan 18, 2019 at 1:51 AM Udi Meiri <eh...@google.com> wrote:
>> >
>> > Hi,
>> > I'd like to add support for creating files using a cloud Key Management System.
>> > A KMS allows you to audit, create, rotate, and disable encryption keys. Both AWS and GCP have such a service..
>> >
>> > I wanted to show the community what I've been working on and see if there are any comments or objection before submitting a PR.
>> > https://github.com/udim/beam/commit/d29f1ef26c58489416a2d413eb029596d96e1f25
>> >
>> > Reference docs:
>> > AWS S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
>> > GCP GCS: https://cloud.google.com/storage/docs/encryption/using-customer-managed-keys#add-object-key

Re: Adding KMS support to generic filesystem interface

Posted by Udi Meiri <eh...@google.com>.
Hi Ismaël,
I'd like your feedback, especially from the AWS perspective.
I wasn't aware of BEAM-3821, but I did create a JIRA for Cloud KMS support
on GCS: https://issues.apache.org/jira/browse/BEAM-5959

Some details of my plan for KMS support:
1. Add KMS settings to sources and sinks.
2. Add a --kmsKey flag that is passed to the runner and applies to pipeline
state.

On Fri, Jan 18, 2019 at 8:24 AM Ismaël Mejía <ie...@gmail.com> wrote:

> Hello Udi,
>
> I implemented the support for KMS in Amazon and I am really interested
> in check your PR. However I won't have time to do it until next
> monday. I hope waiting a bit is ok with you if you want some feedback
> from me.
>
> I am curious if you considered or are aware of this issue:
> BEAM-3821 Support a pluggable key management system (KMS)
> https://issues.apache.org/jira/browse/BEAM-3821
>
>
> On Fri, Jan 18, 2019 at 1:51 AM Udi Meiri <eh...@google.com> wrote:
> >
> > Hi,
> > I'd like to add support for creating files using a cloud Key Management
> System.
> > A KMS allows you to audit, create, rotate, and disable encryption keys.
> Both AWS and GCP have such a service..
> >
> > I wanted to show the community what I've been working on and see if
> there are any comments or objection before submitting a PR.
> >
> https://github.com/udim/beam/commit/d29f1ef26c58489416a2d413eb029596d96e1f25
> >
> > Reference docs:
> > AWS S3:
> https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> > GCP GCS:
> https://cloud.google.com/storage/docs/encryption/using-customer-managed-keys#add-object-key
>

Re: Adding KMS support to generic filesystem interface

Posted by Ismaël Mejía <ie...@gmail.com>.
Hello Udi,

I implemented the support for KMS in Amazon and I am really interested
in check your PR. However I won't have time to do it until next
monday. I hope waiting a bit is ok with you if you want some feedback
from me.

I am curious if you considered or are aware of this issue:
BEAM-3821 Support a pluggable key management system (KMS)
https://issues.apache.org/jira/browse/BEAM-3821


On Fri, Jan 18, 2019 at 1:51 AM Udi Meiri <eh...@google.com> wrote:
>
> Hi,
> I'd like to add support for creating files using a cloud Key Management System.
> A KMS allows you to audit, create, rotate, and disable encryption keys. Both AWS and GCP have such a service..
>
> I wanted to show the community what I've been working on and see if there are any comments or objection before submitting a PR.
> https://github.com/udim/beam/commit/d29f1ef26c58489416a2d413eb029596d96e1f25
>
> Reference docs:
> AWS S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> GCP GCS: https://cloud.google.com/storage/docs/encryption/using-customer-managed-keys#add-object-key