Write to S3 with server side encryption in KMS mode

[Nisrina Luthfiyati <ni...@gmail.com>]

Hi all,

I'm trying to save a spark application output to a bucket in S3. The data
is supposed to be encrypted with S3's server side encryption using KMS
mode, which typically (using java api/cli) would require us to pass the
sse-kms key when writing the data. I currently have not found a way to do
this using spark hadoop config. Would anyone have any idea how this can be
done or whether this is possible?

Thanks,
Nisrina.

Re: Write to S3 with server side encryption in KMS mode

[Nisrina Luthfiyati <ni...@gmail.com>]

Ah, alright then, that looks like that's the case. Thank you for the info.
I'm probably going to try to use the s3 managed encryption, from what I
read this is supported by setting fs.s3a.server-side-encryption-algorithm
parameter.

Thanks!
Nisrina

On Tue, Jan 26, 2016 at 11:55 PM, Ewan Leith <ew...@realitymine.com>
wrote:

> Hi Nisrina, I’m not aware of any support for KMS keys in s3n, s3a or the
> EMR specific EMRFS s3 driver.
>
>
>
> If you’re using EMRFS with Amazon’s EMR, you can use KMS keys with
> client-side encryption
>
>
>
>
> http://docs.aws.amazon.com/kms/latest/developerguide/services-emr.html#emrfs-encrypt
>
>
>
> If this has changed, I’d love to know, but I’m pretty sure it hasn’t.
>
>
>
> The alternative is to write to HDFS, then copy the data across in bulk.
>
>
>
> Thanks,
>
> Ewan
>
>
>
>
>
>
>
> *From:* Nisrina Luthfiyati [mailto:nisrina.luthfiyati@gmail.com]
> *Sent:* 26 January 2016 10:42
> *To:* user <us...@spark.apache.org>
> *Subject:* Write to S3 with server side encryption in KMS mode
>
>
>
> Hi all,
>
>
>
> I'm trying to save a spark application output to a bucket in S3. The data
> is supposed to be encrypted with S3's server side encryption using KMS
> mode, which typically (using java api/cli) would require us to pass the
> sse-kms key when writing the data. I currently have not found a way to do
> this using spark hadoop config. Would anyone have any idea how this can be
> done or whether this is possible?
>
>
>
> Thanks,
>
> Nisrina.
>
>
>



-- 
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina

RE: Write to S3 with server side encryption in KMS mode

[Ewan Leith <ew...@realitymine.com>]

Hi Nisrina, I’m not aware of any support for KMS keys in s3n, s3a or the EMR specific EMRFS s3 driver.

If you’re using EMRFS with Amazon’s EMR, you can use KMS keys with client-side encryption

http://docs.aws.amazon.com/kms/latest/developerguide/services-emr.html#emrfs-encrypt

If this has changed, I’d love to know, but I’m pretty sure it hasn’t.

The alternative is to write to HDFS, then copy the data across in bulk.

Thanks,
Ewan



From: Nisrina Luthfiyati [mailto:nisrina.luthfiyati@gmail.com]
Sent: 26 January 2016 10:42
To: user <us...@spark.apache.org>
Subject: Write to S3 with server side encryption in KMS mode

Hi all,

I'm trying to save a spark application output to a bucket in S3. The data is supposed to be encrypted with S3's server side encryption using KMS mode, which typically (using java api/cli) would require us to pass the sse-kms key when writing the data. I currently have not found a way to do this using spark hadoop config. Would anyone have any idea how this can be done or whether this is possible?

Thanks,
Nisrina.