You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Plamen Jeliazkov <pl...@wandisco.com> on 2015/02/18 01:04:57 UTC
Encryption At Rest Question
Hey guys,
I had a question about how the new file encryption work done primarily in
HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
--
5 reasons your Hadoop needs WANdisco
<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND
<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
If you are not the intended recipient, please notify us immediately and
destroy the message without disclosing its contents to anyone. Any
distribution, use or copying of this e-mail or the information it contains
by other than an intended recipient is unauthorized. The views and
opinions expressed in this e-mail message are the author's own and may not
reflect the views and opinions of WANdisco, unless the author is authorized
by WANdisco to express such views or opinions on its behalf. All email
sent to or from this address is subject to electronic storage and review by
WANdisco. Although WANdisco operates anti-virus programs, it does not
accept responsibility for any damage whatsoever caused by viruses being
passed.
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you all for clarifying.
Indeed both the ways 1) via the contents of file block and 2) looking at
the /.reserved/raw/.. confirms the file is encrypted.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 6:58 PM, Charles Lamb <cl...@cloudera.com> wrote:
> On 2/24/2015 8:56 PM, Liu, Yi A wrote:
>
>> The data is decrypted on client side after obtaining DEK from KMS, *not*
>> decrypted by DN.
>>
> My colleague Yi is correct that data is not decrypted by the DN with one
> exception: WebHDFS uses the DN as the proxy and therefore the DN does the
> decryption in that case. HttpFs is recommended instead.
>
>> Right, currently DEK is better to be protected by https on the wire.
>>
>> If you want to confirm the file is encrypted, one way is to see the
>> content of file blocks.
>>
> Another way is to use the /.reserved/raw prefix on a file. This special
> prefix is only accessible by the hdfs admin. It gives the encrypted (raw)
> bits of a file rather than the decrypted bits. For example, if you have a
> file /ez/myfile, then /.reserved/raw/ez/myfile will yield the encrypted
> bits of the file.
>
> Charles
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you all for clarifying.
Indeed both the ways 1) via the contents of file block and 2) looking at
the /.reserved/raw/.. confirms the file is encrypted.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 6:58 PM, Charles Lamb <cl...@cloudera.com> wrote:
> On 2/24/2015 8:56 PM, Liu, Yi A wrote:
>
>> The data is decrypted on client side after obtaining DEK from KMS, *not*
>> decrypted by DN.
>>
> My colleague Yi is correct that data is not decrypted by the DN with one
> exception: WebHDFS uses the DN as the proxy and therefore the DN does the
> decryption in that case. HttpFs is recommended instead.
>
>> Right, currently DEK is better to be protected by https on the wire.
>>
>> If you want to confirm the file is encrypted, one way is to see the
>> content of file blocks.
>>
> Another way is to use the /.reserved/raw prefix on a file. This special
> prefix is only accessible by the hdfs admin. It gives the encrypted (raw)
> bits of a file rather than the decrypted bits. For example, if you have a
> file /ez/myfile, then /.reserved/raw/ez/myfile will yield the encrypted
> bits of the file.
>
> Charles
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you all for clarifying.
Indeed both the ways 1) via the contents of file block and 2) looking at
the /.reserved/raw/.. confirms the file is encrypted.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 6:58 PM, Charles Lamb <cl...@cloudera.com> wrote:
> On 2/24/2015 8:56 PM, Liu, Yi A wrote:
>
>> The data is decrypted on client side after obtaining DEK from KMS, *not*
>> decrypted by DN.
>>
> My colleague Yi is correct that data is not decrypted by the DN with one
> exception: WebHDFS uses the DN as the proxy and therefore the DN does the
> decryption in that case. HttpFs is recommended instead.
>
>> Right, currently DEK is better to be protected by https on the wire.
>>
>> If you want to confirm the file is encrypted, one way is to see the
>> content of file blocks.
>>
> Another way is to use the /.reserved/raw prefix on a file. This special
> prefix is only accessible by the hdfs admin. It gives the encrypted (raw)
> bits of a file rather than the decrypted bits. For example, if you have a
> file /ez/myfile, then /.reserved/raw/ez/myfile will yield the encrypted
> bits of the file.
>
> Charles
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you all for clarifying.
Indeed both the ways 1) via the contents of file block and 2) looking at
the /.reserved/raw/.. confirms the file is encrypted.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 6:58 PM, Charles Lamb <cl...@cloudera.com> wrote:
> On 2/24/2015 8:56 PM, Liu, Yi A wrote:
>
>> The data is decrypted on client side after obtaining DEK from KMS, *not*
>> decrypted by DN.
>>
> My colleague Yi is correct that data is not decrypted by the DN with one
> exception: WebHDFS uses the DN as the proxy and therefore the DN does the
> decryption in that case. HttpFs is recommended instead.
>
>> Right, currently DEK is better to be protected by https on the wire.
>>
>> If you want to confirm the file is encrypted, one way is to see the
>> content of file blocks.
>>
> Another way is to use the /.reserved/raw prefix on a file. This special
> prefix is only accessible by the hdfs admin. It gives the encrypted (raw)
> bits of a file rather than the decrypted bits. For example, if you have a
> file /ez/myfile, then /.reserved/raw/ez/myfile will yield the encrypted
> bits of the file.
>
> Charles
>
>
Re: Encryption At Rest Question
Posted by Charles Lamb <cl...@cloudera.com>.
On 2/24/2015 8:56 PM, Liu, Yi A wrote:
> The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
My colleague Yi is correct that data is not decrypted by the DN with one
exception: WebHDFS uses the DN as the proxy and therefore the DN does
the decryption in that case. HttpFs is recommended instead.
> Right, currently DEK is better to be protected by https on the wire.
>
> If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Another way is to use the /.reserved/raw prefix on a file. This special
prefix is only accessible by the hdfs admin. It gives the encrypted
(raw) bits of a file rather than the decrypted bits. For example, if you
have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the
encrypted bits of the file.
Charles
Re: Encryption At Rest Question
Posted by Charles Lamb <cl...@cloudera.com>.
On 2/24/2015 8:56 PM, Liu, Yi A wrote:
> The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
My colleague Yi is correct that data is not decrypted by the DN with one
exception: WebHDFS uses the DN as the proxy and therefore the DN does
the decryption in that case. HttpFs is recommended instead.
> Right, currently DEK is better to be protected by https on the wire.
>
> If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Another way is to use the /.reserved/raw prefix on a file. This special
prefix is only accessible by the hdfs admin. It gives the encrypted
(raw) bits of a file rather than the decrypted bits. For example, if you
have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the
encrypted bits of the file.
Charles
Re: Encryption At Rest Question
Posted by Charles Lamb <cl...@cloudera.com>.
On 2/24/2015 8:56 PM, Liu, Yi A wrote:
> The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
My colleague Yi is correct that data is not decrypted by the DN with one
exception: WebHDFS uses the DN as the proxy and therefore the DN does
the decryption in that case. HttpFs is recommended instead.
> Right, currently DEK is better to be protected by https on the wire.
>
> If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Another way is to use the /.reserved/raw prefix on a file. This special
prefix is only accessible by the hdfs admin. It gives the encrypted
(raw) bits of a file rather than the decrypted bits. For example, if you
have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the
encrypted bits of the file.
Charles
Re: Encryption At Rest Question
Posted by Charles Lamb <cl...@cloudera.com>.
On 2/24/2015 8:56 PM, Liu, Yi A wrote:
> The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
My colleague Yi is correct that data is not decrypted by the DN with one
exception: WebHDFS uses the DN as the proxy and therefore the DN does
the decryption in that case. HttpFs is recommended instead.
> Right, currently DEK is better to be protected by https on the wire.
>
> If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Another way is to use the /.reserved/raw prefix on a file. This special
prefix is only accessible by the hdfs admin. It gives the encrypted
(raw) bits of a file rather than the decrypted bits. For example, if you
have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the
encrypted bits of the file.
Charles
Re: Encryption At Rest Question
Posted by Charles Lamb <cl...@cloudera.com>.
On 2/24/2015 8:56 PM, Liu, Yi A wrote:
> The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
My colleague Yi is correct that data is not decrypted by the DN with one
exception: WebHDFS uses the DN as the proxy and therefore the DN does
the decryption in that case. HttpFs is recommended instead.
> Right, currently DEK is better to be protected by https on the wire.
>
> If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Another way is to use the /.reserved/raw prefix on a file. This special
prefix is only accessible by the hdfs admin. It gives the encrypted
(raw) bits of a file rather than the decrypted bits. For example, if you
have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the
encrypted bits of the file.
Charles
RE: Encryption At Rest Question
Posted by "Liu, Yi A" <yi...@intel.com>.
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
Right, currently DEK is better to be protected by https on the wire.
If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Regards,
Yi Liu
From: Rajesh Kartha [mailto:kartha02@gmail.com]
Sent: Wednesday, February 25, 2015 3:48 AM
To: user@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
RE: Encryption At Rest Question
Posted by "Liu, Yi A" <yi...@intel.com>.
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
Right, currently DEK is better to be protected by https on the wire.
If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Regards,
Yi Liu
From: Rajesh Kartha [mailto:kartha02@gmail.com]
Sent: Wednesday, February 25, 2015 3:48 AM
To: user@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
RE: Encryption At Rest Question
Posted by "Liu, Yi A" <yi...@intel.com>.
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
Right, currently DEK is better to be protected by https on the wire.
If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Regards,
Yi Liu
From: Rajesh Kartha [mailto:kartha02@gmail.com]
Sent: Wednesday, February 25, 2015 3:48 AM
To: user@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can set some ACL on the KMS so you could have an open access from an HDFS ACL perspective but a restriction from KSM side.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 22:16
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>> wrote:
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can set some ACL on the KMS so you could have an open access from an HDFS ACL perspective but a restriction from KSM side.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 22:16
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>> wrote:
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can set some ACL on the KMS so you could have an open access from an HDFS ACL perspective but a restriction from KSM side.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 22:16
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>> wrote:
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can set some ACL on the KMS so you could have an open access from an HDFS ACL perspective but a restriction from KSM side.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 22:16
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>> wrote:
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for
other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second
approach you suggested. With hadoop fsck /mysecureDir -files -blocks
-locations get the blocks for the directory, then go to the data node and
perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>
wrote:
> You can try looking at it with a user who doesn’t have permission to
> the folder. An alternative is to check which block it is on Linux and
> looking at the block using cat from a linux shell.
>
> Olivier
>
>
> From: Rajesh Kartha <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Tuesday, 24 February 2015 19:47
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: Encryption At Rest Question
>
> I was trying out the Transparent data at rest encryption and was able
> to setup the KMS, zones etc. and add
> files to the zone.
>
> How do I confirm if the files I added to the encryption zone are
> encrypted ? Is there a way to view
> the raw file, a *hdfs fs -cat *shows me the actual contents of the files
> since the datanode decrypts it
> before sending it.
>
> Thanks,
> Rajesh
>
>
> On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
> wrote:
>
>> In case of SSL enabled cluster, the DEK will be encrypted on the wire
>> by the SSL layer.
>>
>> In case of non-SSL enabled cluster, it is not. But the intercepter only
>> gets the DEK and not the encrypted data, so the data is still safe. Only if
>> the intercepter also manages to gain access to the encrypted data block and
>> associate that with the corresponding DEK, then the data is compromised.
>> Given that each HDFS file has a different DEK, the intercepter has to gain
>> quite a bit of access before the data is compromised.
>>
>> On 18 February 2015 at 00:04, Plamen Jeliazkov <
>> plamen.jeliazkov@wandisco.com> wrote:
>>
>>> Hey guys,
>>>
>>> I had a question about how the new file encryption work done primarily
>>> in HDFS-6134.
>>>
>>> I was just curious, how is the DEK protected on the wire?
>>> Particularly after the KMS decrypts the EDEK and returns it to the
>>> client.
>>>
>>> Thanks,
>>> -Plamen
>>>
>>>
>>>
>>> 5 reasons your Hadoop needs WANdisco
>>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>>
>>> Listed on the London Stock Exchange: WAND
>>> <http://www.bloomberg.com/quote/WAND:LN>
>>>
>>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>>> privilege. If you are not the intended recipient, please notify us
>>> immediately and destroy the message without disclosing its contents to
>>> anyone. Any distribution, use or copying of this e-mail or the information
>>> it contains by other than an intended recipient is unauthorized. The views
>>> and opinions expressed in this e-mail message are the author's own and may
>>> not reflect the views and opinions of WANdisco, unless the author is
>>> authorized by WANdisco to express such views or opinions on its behalf.
>>> All email sent to or from this address is subject to electronic storage and
>>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>>> does not accept responsibility for any damage whatsoever caused by viruses
>>> being passed.
>>>
>>
>>
>>
>> --
>> Regards,
>> Ranadip Chatterjee
>>
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for
other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second
approach you suggested. With hadoop fsck /mysecureDir -files -blocks
-locations get the blocks for the directory, then go to the data node and
perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>
wrote:
> You can try looking at it with a user who doesn’t have permission to
> the folder. An alternative is to check which block it is on Linux and
> looking at the block using cat from a linux shell.
>
> Olivier
>
>
> From: Rajesh Kartha <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Tuesday, 24 February 2015 19:47
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: Encryption At Rest Question
>
> I was trying out the Transparent data at rest encryption and was able
> to setup the KMS, zones etc. and add
> files to the zone.
>
> How do I confirm if the files I added to the encryption zone are
> encrypted ? Is there a way to view
> the raw file, a *hdfs fs -cat *shows me the actual contents of the files
> since the datanode decrypts it
> before sending it.
>
> Thanks,
> Rajesh
>
>
> On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
> wrote:
>
>> In case of SSL enabled cluster, the DEK will be encrypted on the wire
>> by the SSL layer.
>>
>> In case of non-SSL enabled cluster, it is not. But the intercepter only
>> gets the DEK and not the encrypted data, so the data is still safe. Only if
>> the intercepter also manages to gain access to the encrypted data block and
>> associate that with the corresponding DEK, then the data is compromised.
>> Given that each HDFS file has a different DEK, the intercepter has to gain
>> quite a bit of access before the data is compromised.
>>
>> On 18 February 2015 at 00:04, Plamen Jeliazkov <
>> plamen.jeliazkov@wandisco.com> wrote:
>>
>>> Hey guys,
>>>
>>> I had a question about how the new file encryption work done primarily
>>> in HDFS-6134.
>>>
>>> I was just curious, how is the DEK protected on the wire?
>>> Particularly after the KMS decrypts the EDEK and returns it to the
>>> client.
>>>
>>> Thanks,
>>> -Plamen
>>>
>>>
>>>
>>> 5 reasons your Hadoop needs WANdisco
>>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>>
>>> Listed on the London Stock Exchange: WAND
>>> <http://www.bloomberg.com/quote/WAND:LN>
>>>
>>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>>> privilege. If you are not the intended recipient, please notify us
>>> immediately and destroy the message without disclosing its contents to
>>> anyone. Any distribution, use or copying of this e-mail or the information
>>> it contains by other than an intended recipient is unauthorized. The views
>>> and opinions expressed in this e-mail message are the author's own and may
>>> not reflect the views and opinions of WANdisco, unless the author is
>>> authorized by WANdisco to express such views or opinions on its behalf.
>>> All email sent to or from this address is subject to electronic storage and
>>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>>> does not accept responsibility for any damage whatsoever caused by viruses
>>> being passed.
>>>
>>
>>
>>
>> --
>> Regards,
>> Ranadip Chatterjee
>>
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for
other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second
approach you suggested. With hadoop fsck /mysecureDir -files -blocks
-locations get the blocks for the directory, then go to the data node and
perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>
wrote:
> You can try looking at it with a user who doesn’t have permission to
> the folder. An alternative is to check which block it is on Linux and
> looking at the block using cat from a linux shell.
>
> Olivier
>
>
> From: Rajesh Kartha <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Tuesday, 24 February 2015 19:47
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: Encryption At Rest Question
>
> I was trying out the Transparent data at rest encryption and was able
> to setup the KMS, zones etc. and add
> files to the zone.
>
> How do I confirm if the files I added to the encryption zone are
> encrypted ? Is there a way to view
> the raw file, a *hdfs fs -cat *shows me the actual contents of the files
> since the datanode decrypts it
> before sending it.
>
> Thanks,
> Rajesh
>
>
> On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
> wrote:
>
>> In case of SSL enabled cluster, the DEK will be encrypted on the wire
>> by the SSL layer.
>>
>> In case of non-SSL enabled cluster, it is not. But the intercepter only
>> gets the DEK and not the encrypted data, so the data is still safe. Only if
>> the intercepter also manages to gain access to the encrypted data block and
>> associate that with the corresponding DEK, then the data is compromised.
>> Given that each HDFS file has a different DEK, the intercepter has to gain
>> quite a bit of access before the data is compromised.
>>
>> On 18 February 2015 at 00:04, Plamen Jeliazkov <
>> plamen.jeliazkov@wandisco.com> wrote:
>>
>>> Hey guys,
>>>
>>> I had a question about how the new file encryption work done primarily
>>> in HDFS-6134.
>>>
>>> I was just curious, how is the DEK protected on the wire?
>>> Particularly after the KMS decrypts the EDEK and returns it to the
>>> client.
>>>
>>> Thanks,
>>> -Plamen
>>>
>>>
>>>
>>> 5 reasons your Hadoop needs WANdisco
>>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>>
>>> Listed on the London Stock Exchange: WAND
>>> <http://www.bloomberg.com/quote/WAND:LN>
>>>
>>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>>> privilege. If you are not the intended recipient, please notify us
>>> immediately and destroy the message without disclosing its contents to
>>> anyone. Any distribution, use or copying of this e-mail or the information
>>> it contains by other than an intended recipient is unauthorized. The views
>>> and opinions expressed in this e-mail message are the author's own and may
>>> not reflect the views and opinions of WANdisco, unless the author is
>>> authorized by WANdisco to express such views or opinions on its behalf.
>>> All email sent to or from this address is subject to electronic storage and
>>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>>> does not accept responsibility for any damage whatsoever caused by viruses
>>> being passed.
>>>
>>
>>
>>
>> --
>> Regards,
>> Ranadip Chatterjee
>>
>
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
Thank you Olivier,
I suppose with the first suggestion - locking the dir to be unreadable for
other users, the HDFS permissions would
kick in and prevent an unwarranted user to read them.
However, I wanted to see the actual encrypted data so I used the second
approach you suggested. With hadoop fsck /mysecureDir -files -blocks
-locations get the blocks for the directory, then go to the data node and
perform a cat to see cryptic data for those block.
Regards,
Rajesh
On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault <or...@hortonworks.com>
wrote:
> You can try looking at it with a user who doesn’t have permission to
> the folder. An alternative is to check which block it is on Linux and
> looking at the block using cat from a linux shell.
>
> Olivier
>
>
> From: Rajesh Kartha <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Tuesday, 24 February 2015 19:47
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: Encryption At Rest Question
>
> I was trying out the Transparent data at rest encryption and was able
> to setup the KMS, zones etc. and add
> files to the zone.
>
> How do I confirm if the files I added to the encryption zone are
> encrypted ? Is there a way to view
> the raw file, a *hdfs fs -cat *shows me the actual contents of the files
> since the datanode decrypts it
> before sending it.
>
> Thanks,
> Rajesh
>
>
> On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
> wrote:
>
>> In case of SSL enabled cluster, the DEK will be encrypted on the wire
>> by the SSL layer.
>>
>> In case of non-SSL enabled cluster, it is not. But the intercepter only
>> gets the DEK and not the encrypted data, so the data is still safe. Only if
>> the intercepter also manages to gain access to the encrypted data block and
>> associate that with the corresponding DEK, then the data is compromised.
>> Given that each HDFS file has a different DEK, the intercepter has to gain
>> quite a bit of access before the data is compromised.
>>
>> On 18 February 2015 at 00:04, Plamen Jeliazkov <
>> plamen.jeliazkov@wandisco.com> wrote:
>>
>>> Hey guys,
>>>
>>> I had a question about how the new file encryption work done primarily
>>> in HDFS-6134.
>>>
>>> I was just curious, how is the DEK protected on the wire?
>>> Particularly after the KMS decrypts the EDEK and returns it to the
>>> client.
>>>
>>> Thanks,
>>> -Plamen
>>>
>>>
>>>
>>> 5 reasons your Hadoop needs WANdisco
>>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>>
>>> Listed on the London Stock Exchange: WAND
>>> <http://www.bloomberg.com/quote/WAND:LN>
>>>
>>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>>> privilege. If you are not the intended recipient, please notify us
>>> immediately and destroy the message without disclosing its contents to
>>> anyone. Any distribution, use or copying of this e-mail or the information
>>> it contains by other than an intended recipient is unauthorized. The views
>>> and opinions expressed in this e-mail message are the author's own and may
>>> not reflect the views and opinions of WANdisco, unless the author is
>>> authorized by WANdisco to express such views or opinions on its behalf.
>>> All email sent to or from this address is subject to electronic storage and
>>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>>> does not accept responsibility for any damage whatsoever caused by viruses
>>> being passed.
>>>
>>
>>
>>
>> --
>> Regards,
>> Ranadip Chatterjee
>>
>
>
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Olivier Renault <or...@hortonworks.com>.
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell.
Olivier
From: Rajesh Kartha <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, 24 February 2015 19:47
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
RE: Encryption At Rest Question
Posted by "Liu, Yi A" <yi...@intel.com>.
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
Right, currently DEK is better to be protected by https on the wire.
If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Regards,
Yi Liu
From: Rajesh Kartha [mailto:kartha02@gmail.com]
Sent: Wednesday, February 25, 2015 3:48 AM
To: user@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
RE: Encryption At Rest Question
Posted by "Liu, Yi A" <yi...@intel.com>.
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN.
Right, currently DEK is better to be protected by https on the wire.
If you want to confirm the file is encrypted, one way is to see the content of file blocks.
Regards,
Yi Liu
From: Rajesh Kartha [mailto:kartha02@gmail.com]
Sent: Wednesday, February 25, 2015 3:48 AM
To: user@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <pl...@wandisco.com>> wrote:
Hey guys,
I had a question about how the new file encryption work done primarily in HDFS-6134.
I was just curious, how is the DEK protected on the wire?
Particularly after the KMS decrypts the EDEK and returns it to the client.
Thanks,
-Plamen
5 reasons your Hadoop needs WANdisco<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN>
THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
I was trying out the Transparent data at rest encryption and was able to
setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted
? Is there a way to view
the raw file, a *hdfs fs -cat *shows me the actual contents of the files
since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> In case of SSL enabled cluster, the DEK will be encrypted on the wire by
> the SSL layer.
>
> In case of non-SSL enabled cluster, it is not. But the intercepter only
> gets the DEK and not the encrypted data, so the data is still safe. Only if
> the intercepter also manages to gain access to the encrypted data block and
> associate that with the corresponding DEK, then the data is compromised.
> Given that each HDFS file has a different DEK, the intercepter has to gain
> quite a bit of access before the data is compromised.
>
> On 18 February 2015 at 00:04, Plamen Jeliazkov <
> plamen.jeliazkov@wandisco.com> wrote:
>
>> Hey guys,
>>
>> I had a question about how the new file encryption work done primarily in
>> HDFS-6134.
>>
>> I was just curious, how is the DEK protected on the wire?
>> Particularly after the KMS decrypts the EDEK and returns it to the client.
>>
>> Thanks,
>> -Plamen
>>
>>
>>
>> 5 reasons your Hadoop needs WANdisco
>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>
>> Listed on the London Stock Exchange: WAND
>> <http://www.bloomberg.com/quote/WAND:LN>
>>
>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>> privilege. If you are not the intended recipient, please notify us
>> immediately and destroy the message without disclosing its contents to
>> anyone. Any distribution, use or copying of this e-mail or the information
>> it contains by other than an intended recipient is unauthorized. The views
>> and opinions expressed in this e-mail message are the author's own and may
>> not reflect the views and opinions of WANdisco, unless the author is
>> authorized by WANdisco to express such views or opinions on its behalf.
>> All email sent to or from this address is subject to electronic storage and
>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>> does not accept responsibility for any damage whatsoever caused by viruses
>> being passed.
>>
>
>
>
> --
> Regards,
> Ranadip Chatterjee
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
I was trying out the Transparent data at rest encryption and was able to
setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted
? Is there a way to view
the raw file, a *hdfs fs -cat *shows me the actual contents of the files
since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> In case of SSL enabled cluster, the DEK will be encrypted on the wire by
> the SSL layer.
>
> In case of non-SSL enabled cluster, it is not. But the intercepter only
> gets the DEK and not the encrypted data, so the data is still safe. Only if
> the intercepter also manages to gain access to the encrypted data block and
> associate that with the corresponding DEK, then the data is compromised.
> Given that each HDFS file has a different DEK, the intercepter has to gain
> quite a bit of access before the data is compromised.
>
> On 18 February 2015 at 00:04, Plamen Jeliazkov <
> plamen.jeliazkov@wandisco.com> wrote:
>
>> Hey guys,
>>
>> I had a question about how the new file encryption work done primarily in
>> HDFS-6134.
>>
>> I was just curious, how is the DEK protected on the wire?
>> Particularly after the KMS decrypts the EDEK and returns it to the client.
>>
>> Thanks,
>> -Plamen
>>
>>
>>
>> 5 reasons your Hadoop needs WANdisco
>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>
>> Listed on the London Stock Exchange: WAND
>> <http://www.bloomberg.com/quote/WAND:LN>
>>
>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>> privilege. If you are not the intended recipient, please notify us
>> immediately and destroy the message without disclosing its contents to
>> anyone. Any distribution, use or copying of this e-mail or the information
>> it contains by other than an intended recipient is unauthorized. The views
>> and opinions expressed in this e-mail message are the author's own and may
>> not reflect the views and opinions of WANdisco, unless the author is
>> authorized by WANdisco to express such views or opinions on its behalf.
>> All email sent to or from this address is subject to electronic storage and
>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>> does not accept responsibility for any damage whatsoever caused by viruses
>> being passed.
>>
>
>
>
> --
> Regards,
> Ranadip Chatterjee
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
I was trying out the Transparent data at rest encryption and was able to
setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted
? Is there a way to view
the raw file, a *hdfs fs -cat *shows me the actual contents of the files
since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> In case of SSL enabled cluster, the DEK will be encrypted on the wire by
> the SSL layer.
>
> In case of non-SSL enabled cluster, it is not. But the intercepter only
> gets the DEK and not the encrypted data, so the data is still safe. Only if
> the intercepter also manages to gain access to the encrypted data block and
> associate that with the corresponding DEK, then the data is compromised.
> Given that each HDFS file has a different DEK, the intercepter has to gain
> quite a bit of access before the data is compromised.
>
> On 18 February 2015 at 00:04, Plamen Jeliazkov <
> plamen.jeliazkov@wandisco.com> wrote:
>
>> Hey guys,
>>
>> I had a question about how the new file encryption work done primarily in
>> HDFS-6134.
>>
>> I was just curious, how is the DEK protected on the wire?
>> Particularly after the KMS decrypts the EDEK and returns it to the client.
>>
>> Thanks,
>> -Plamen
>>
>>
>>
>> 5 reasons your Hadoop needs WANdisco
>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>
>> Listed on the London Stock Exchange: WAND
>> <http://www.bloomberg.com/quote/WAND:LN>
>>
>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>> privilege. If you are not the intended recipient, please notify us
>> immediately and destroy the message without disclosing its contents to
>> anyone. Any distribution, use or copying of this e-mail or the information
>> it contains by other than an intended recipient is unauthorized. The views
>> and opinions expressed in this e-mail message are the author's own and may
>> not reflect the views and opinions of WANdisco, unless the author is
>> authorized by WANdisco to express such views or opinions on its behalf.
>> All email sent to or from this address is subject to electronic storage and
>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>> does not accept responsibility for any damage whatsoever caused by viruses
>> being passed.
>>
>
>
>
> --
> Regards,
> Ranadip Chatterjee
>
Re: Encryption At Rest Question
Posted by Rajesh Kartha <ka...@gmail.com>.
I was trying out the Transparent data at rest encryption and was able to
setup the KMS, zones etc. and add
files to the zone.
How do I confirm if the files I added to the encryption zone are encrypted
? Is there a way to view
the raw file, a *hdfs fs -cat *shows me the actual contents of the files
since the datanode decrypts it
before sending it.
Thanks,
Rajesh
On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> In case of SSL enabled cluster, the DEK will be encrypted on the wire by
> the SSL layer.
>
> In case of non-SSL enabled cluster, it is not. But the intercepter only
> gets the DEK and not the encrypted data, so the data is still safe. Only if
> the intercepter also manages to gain access to the encrypted data block and
> associate that with the corresponding DEK, then the data is compromised.
> Given that each HDFS file has a different DEK, the intercepter has to gain
> quite a bit of access before the data is compromised.
>
> On 18 February 2015 at 00:04, Plamen Jeliazkov <
> plamen.jeliazkov@wandisco.com> wrote:
>
>> Hey guys,
>>
>> I had a question about how the new file encryption work done primarily in
>> HDFS-6134.
>>
>> I was just curious, how is the DEK protected on the wire?
>> Particularly after the KMS decrypts the EDEK and returns it to the client.
>>
>> Thanks,
>> -Plamen
>>
>>
>>
>> 5 reasons your Hadoop needs WANdisco
>> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>>
>> Listed on the London Stock Exchange: WAND
>> <http://www.bloomberg.com/quote/WAND:LN>
>>
>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY
>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
>> subsidiaries, ("WANdisco") does not waive any confidentiality or
>> privilege. If you are not the intended recipient, please notify us
>> immediately and destroy the message without disclosing its contents to
>> anyone. Any distribution, use or copying of this e-mail or the information
>> it contains by other than an intended recipient is unauthorized. The views
>> and opinions expressed in this e-mail message are the author's own and may
>> not reflect the views and opinions of WANdisco, unless the author is
>> authorized by WANdisco to express such views or opinions on its behalf.
>> All email sent to or from this address is subject to electronic storage and
>> review by WANdisco. Although WANdisco operates anti-virus programs, it
>> does not accept responsibility for any damage whatsoever caused by viruses
>> being passed.
>>
>
>
>
> --
> Regards,
> Ranadip Chatterjee
>
Re: Encryption At Rest Question
Posted by Ranadip Chatterjee <ra...@gmail.com>.
In case of SSL enabled cluster, the DEK will be encrypted on the wire by
the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only
gets the DEK and not the encrypted data, so the data is still safe. Only if
the intercepter also manages to gain access to the encrypted data block and
associate that with the corresponding DEK, then the data is compromised.
Given that each HDFS file has a different DEK, the intercepter has to gain
quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <
plamen.jeliazkov@wandisco.com> wrote:
> Hey guys,
>
> I had a question about how the new file encryption work done primarily in
> HDFS-6134.
>
> I was just curious, how is the DEK protected on the wire?
> Particularly after the KMS decrypts the EDEK and returns it to the client.
>
> Thanks,
> -Plamen
>
>
>
> 5 reasons your Hadoop needs WANdisco
> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>
> Listed on the London Stock Exchange: WAND
> <http://www.bloomberg.com/quote/WAND:LN>
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege. If you are not the intended recipient, please notify us
> immediately and destroy the message without disclosing its contents to
> anyone. Any distribution, use or copying of this e-mail or the information
> it contains by other than an intended recipient is unauthorized. The views
> and opinions expressed in this e-mail message are the author's own and may
> not reflect the views and opinions of WANdisco, unless the author is
> authorized by WANdisco to express such views or opinions on its behalf.
> All email sent to or from this address is subject to electronic storage and
> review by WANdisco. Although WANdisco operates anti-virus programs, it
> does not accept responsibility for any damage whatsoever caused by viruses
> being passed.
>
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Ranadip Chatterjee <ra...@gmail.com>.
In case of SSL enabled cluster, the DEK will be encrypted on the wire by
the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only
gets the DEK and not the encrypted data, so the data is still safe. Only if
the intercepter also manages to gain access to the encrypted data block and
associate that with the corresponding DEK, then the data is compromised.
Given that each HDFS file has a different DEK, the intercepter has to gain
quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <
plamen.jeliazkov@wandisco.com> wrote:
> Hey guys,
>
> I had a question about how the new file encryption work done primarily in
> HDFS-6134.
>
> I was just curious, how is the DEK protected on the wire?
> Particularly after the KMS decrypts the EDEK and returns it to the client.
>
> Thanks,
> -Plamen
>
>
>
> 5 reasons your Hadoop needs WANdisco
> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>
> Listed on the London Stock Exchange: WAND
> <http://www.bloomberg.com/quote/WAND:LN>
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege. If you are not the intended recipient, please notify us
> immediately and destroy the message without disclosing its contents to
> anyone. Any distribution, use or copying of this e-mail or the information
> it contains by other than an intended recipient is unauthorized. The views
> and opinions expressed in this e-mail message are the author's own and may
> not reflect the views and opinions of WANdisco, unless the author is
> authorized by WANdisco to express such views or opinions on its behalf.
> All email sent to or from this address is subject to electronic storage and
> review by WANdisco. Although WANdisco operates anti-virus programs, it
> does not accept responsibility for any damage whatsoever caused by viruses
> being passed.
>
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Ranadip Chatterjee <ra...@gmail.com>.
In case of SSL enabled cluster, the DEK will be encrypted on the wire by
the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only
gets the DEK and not the encrypted data, so the data is still safe. Only if
the intercepter also manages to gain access to the encrypted data block and
associate that with the corresponding DEK, then the data is compromised.
Given that each HDFS file has a different DEK, the intercepter has to gain
quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <
plamen.jeliazkov@wandisco.com> wrote:
> Hey guys,
>
> I had a question about how the new file encryption work done primarily in
> HDFS-6134.
>
> I was just curious, how is the DEK protected on the wire?
> Particularly after the KMS decrypts the EDEK and returns it to the client.
>
> Thanks,
> -Plamen
>
>
>
> 5 reasons your Hadoop needs WANdisco
> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>
> Listed on the London Stock Exchange: WAND
> <http://www.bloomberg.com/quote/WAND:LN>
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege. If you are not the intended recipient, please notify us
> immediately and destroy the message without disclosing its contents to
> anyone. Any distribution, use or copying of this e-mail or the information
> it contains by other than an intended recipient is unauthorized. The views
> and opinions expressed in this e-mail message are the author's own and may
> not reflect the views and opinions of WANdisco, unless the author is
> authorized by WANdisco to express such views or opinions on its behalf.
> All email sent to or from this address is subject to electronic storage and
> review by WANdisco. Although WANdisco operates anti-virus programs, it
> does not accept responsibility for any damage whatsoever caused by viruses
> being passed.
>
--
Regards,
Ranadip Chatterjee
Re: Encryption At Rest Question
Posted by Ranadip Chatterjee <ra...@gmail.com>.
In case of SSL enabled cluster, the DEK will be encrypted on the wire by
the SSL layer.
In case of non-SSL enabled cluster, it is not. But the intercepter only
gets the DEK and not the encrypted data, so the data is still safe. Only if
the intercepter also manages to gain access to the encrypted data block and
associate that with the corresponding DEK, then the data is compromised.
Given that each HDFS file has a different DEK, the intercepter has to gain
quite a bit of access before the data is compromised.
On 18 February 2015 at 00:04, Plamen Jeliazkov <
plamen.jeliazkov@wandisco.com> wrote:
> Hey guys,
>
> I had a question about how the new file encryption work done primarily in
> HDFS-6134.
>
> I was just curious, how is the DEK protected on the wire?
> Particularly after the KMS decrypts the EDEK and returns it to the client.
>
> Thanks,
> -Plamen
>
>
>
> 5 reasons your Hadoop needs WANdisco
> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>
> Listed on the London Stock Exchange: WAND
> <http://www.bloomberg.com/quote/WAND:LN>
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege. If you are not the intended recipient, please notify us
> immediately and destroy the message without disclosing its contents to
> anyone. Any distribution, use or copying of this e-mail or the information
> it contains by other than an intended recipient is unauthorized. The views
> and opinions expressed in this e-mail message are the author's own and may
> not reflect the views and opinions of WANdisco, unless the author is
> authorized by WANdisco to express such views or opinions on its behalf.
> All email sent to or from this address is subject to electronic storage and
> review by WANdisco. Although WANdisco operates anti-virus programs, it
> does not accept responsibility for any damage whatsoever caused by viruses
> being passed.
>
--
Regards,
Ranadip Chatterjee