You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Gopi Krishnan Nambiar (JIRA)" <ji...@apache.org> on 2019/02/12 19:30:00 UTC
[jira] [Comment Edited] (MAPREDUCE-5890) Support for encrypting
Intermediate data and spills in local filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765567#comment-16765567 ]
Gopi Krishnan Nambiar edited comment on MAPREDUCE-5890 at 2/12/19 7:29 PM:
---------------------------------------------------------------------------
Hello [~vinodkv], [~chris.douglas], [~tucu00], [~asuresh],
Had a question around why this snippet of code was removed (which was added as part of this JIRA - MAPREDUCE-5890) in the File: JobSubmitter.java :
{{int keyLen = CryptoUtils.isShuffleEncrypted(conf)}}? conf.getInt(MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS, MRJobConfig.DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS): SHUFFLE_KEY_LENGTH;
and later reverted and replaced with a constant value:
{{keyGen.init(SHUFFLE_KEY_LENGTH);}}
as part of this change:[https://github.com/apache/hadoop/commit/d9d7bbd99b533da5ca570deb3b8dc8a959c6b4db]
Some context around this question: We are trying to go for FedRamp High Certification and that mandates a key length for HMAC-SHA1 to be at least 112 bits and the current key length is 64 bits. Would be great to know your thoughts on this one.
was (Author: gkrishnan):
Hello [~vinodkv], [~chris.douglas], [~tucu00], [~asuresh],
Had a question around why this snippet of code was removed (which was added as part of this JIRA - MAPREDUCE-5890):
{{int keyLen = CryptoUtils.isShuffleEncrypted(conf)}}? conf.getInt(MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS, MRJobConfig.DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS): SHUFFLE_KEY_LENGTH;
and later reverted and replaced with a constant value:
{{keyGen.init(SHUFFLE_KEY_LENGTH);}}
as part of this change:[https://github.com/apache/hadoop/commit/d9d7bbd99b533da5ca570deb3b8dc8a959c6b4db]
Some context around this question: We are trying to go for FedRamp High Certification and that mandates a key length for HMAC-SHA1 to be at least 112 bits and the current key length is 64 bits. Would be great to know your thoughts on this one.
> Support for encrypting Intermediate data and spills in local filesystem
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-5890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5890
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: security
> Affects Versions: 2.4.0
> Reporter: Alejandro Abdelnur
> Assignee: Arun Suresh
> Priority: Major
> Labels: encryption
> Fix For: 2.6.0, fs-encryption
>
> Attachments: MAPREDUCE-5890.10.patch, MAPREDUCE-5890.11.patch, MAPREDUCE-5890.12.patch, MAPREDUCE-5890.13.patch, MAPREDUCE-5890.14.patch, MAPREDUCE-5890.15.patch, MAPREDUCE-5890.3.patch, MAPREDUCE-5890.4.patch, MAPREDUCE-5890.5.patch, MAPREDUCE-5890.6.patch, MAPREDUCE-5890.7.patch, MAPREDUCE-5890.8.patch, MAPREDUCE-5890.9.patch, org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-output.txt, syslog.tar.gz
>
>
> For some sensitive data, encryption while in flight (network) is not sufficient, it is required that while at rest it should be encrypted. HADOOP-10150 & HDFS-6134 bring encryption at rest for data in filesystem using Hadoop FileSystem API. MapReduce intermediate data and spills should also be encrypted while at rest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org