You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by cn...@apache.org on 2014/10/09 20:51:24 UTC
git commit: HDFS-7195. Update user doc of secure mode about Datanodes
don't require root or jsvc. Contributed by Chris Nauroth.
Repository: hadoop
Updated Branches:
refs/heads/trunk 180afa2f8 -> 909718398
HDFS-7195. Update user doc of secure mode about Datanodes don't require root or jsvc. Contributed by Chris Nauroth.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/90971839
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/90971839
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/90971839
Branch: refs/heads/trunk
Commit: 9097183983cd96ab0fe56b2564d8a63f78b2845c
Parents: 180afa2
Author: cnauroth <cn...@apache.org>
Authored: Thu Oct 9 11:51:12 2014 -0700
Committer: cnauroth <cn...@apache.org>
Committed: Thu Oct 9 11:51:12 2014 -0700
----------------------------------------------------------------------
.../hadoop-common/src/main/conf/hadoop-env.sh | 12 +++--
.../src/site/apt/SecureMode.apt.vm | 47 +++++++++++++++++++-
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 ++
3 files changed, 58 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/90971839/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
index 202098d..b0963af 100644
--- a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
+++ b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
@@ -206,7 +206,10 @@ esac
# custom functions. See hadoop-functions.sh for more information.
#
-# The jsvc implementation to use. Jsvc is required to run secure datanodes.
+# The jsvc implementation to use. Jsvc is required to run secure datanodes
+# that bind to privileged ports to provide authentication of data transfer
+# protocol. Jsvc is not required if SASL is configured for authentication of
+# data transfer protocol using non-privileged ports.
# export JSVC_HOME=/usr/bin
#
@@ -263,8 +266,11 @@ esac
# This is the default:
# export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS"
-# On secure datanodes, user to run the datanode as after dropping privileges
-# This **MUST** be uncommented to enable secure HDFS!
+# On secure datanodes, user to run the datanode as after dropping privileges.
+# This **MUST** be uncommented to enable secure HDFS if using privileged ports
+# to provide authentication of data transfer protocol. This **MUST NOT** be
+# defined if SASL is configured for authentication of data transfer protocol
+# using non-privileged ports.
# export HADOOP_SECURE_DN_USER=hdfs
# Supplemental options for secure datanodes
http://git-wip-us.apache.org/repos/asf/hadoop/blob/90971839/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm
index 54473e6..2085d5b 100644
--- a/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm
+++ b/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm
@@ -244,7 +244,7 @@ KVNO Timestamp Principal
10.222.0.0-15 and 10.113.221.221
can impersonate any user belonging to any group.
- ----
+----
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>10.222.0.0/16,10.113.221.221</value>
@@ -273,6 +273,33 @@ KVNO Timestamp Principal
You must specify <<<HADOOP_SECURE_DN_USER>>> and <<<JSVC_HOME>>>
as environment variables on start up (in hadoop-env.sh).
+ As of version 2.6.0, SASL can be used to authenticate the data transfer
+ protocol. In this configuration, it is no longer required for secured clusters
+ to start the DataNode as root using jsvc and bind to privileged ports. To
+ enable SASL on data transfer protocol, set <<<dfs.data.transfer.protection>>>
+ in hdfs-site.xml, set a non-privileged port for <<<dfs.datanode.address>>>, set
+ <<<dfs.http.policy>>> to <HTTPS_ONLY> and make sure the
+ <<<HADOOP_SECURE_DN_USER>>> environment variable is not defined. Note that it
+ is not possible to use SASL on data transfer protocol if
+ <<<dfs.datanode.address>>> is set to a privileged port. This is required for
+ backwards-compatibility reasons.
+
+ In order to migrate an existing cluster that used root authentication to start
+ using SASL instead, first ensure that version 2.6.0 or later has been deployed
+ to all cluster nodes as well as any external applications that need to connect
+ to the cluster. Only versions 2.6.0 and later of the HDFS client can connect
+ to a DataNode that uses SASL for authentication of data transfer protocol, so
+ it is vital that all callers have the correct version before migrating. After
+ version 2.6.0 or later has been deployed everywhere, update configuration of
+ any external applications to enable SASL. If an HDFS client is enabled for
+ SASL, then it can connect successfully to a DataNode running with either root
+ authentication or SASL authentication. Changing configuration for all clients
+ guarantees that subsequent configuration changes on DataNodes will not disrupt
+ the applications. Finally, each individual DataNode can be migrated by
+ changing its configuration and restarting. It is acceptable to have a mix of
+ some DataNodes running with root authentication and some DataNodes running with
+ SASL authentication temporarily during this migration period, because an HDFS
+ client enabled for SASL can connect to both.
* Data confidentiality
@@ -391,6 +418,10 @@ Configuration for <<<conf/core-site.xml>>>
| <<<dfs.http.policy>>> | <HTTP_ONLY> or <HTTPS_ONLY> or <HTTP_AND_HTTPS> | |
| | | HTTPS_ONLY turns off http access. This option takes precedence over |
| | | the deprecated configuration dfs.https.enable and hadoop.ssl.enabled. |
+| | | If using SASL to authenticate data transfer protocol instead of |
+| | | running DataNode as root and using privileged ports, then this property |
+| | | must be set to <HTTPS_ONLY> to guarantee authentication of HTTP servers. |
+| | | (See <<<dfs.data.transfer.protection>>>.) |
*-------------------------+-------------------------+------------------------+
| <<<dfs.namenode.https-address>>> | <nn_host_fqdn:50470> | |
*-------------------------+-------------------------+------------------------+
@@ -440,6 +471,9 @@ Configuration for <<<conf/hdfs-site.xml>>>
| | | Secure DataNode must use privileged port |
| | | in order to assure that the server was started securely. |
| | | This means that the server must be started via jsvc. |
+| | | Alternatively, this must be set to a non-privileged port if using SASL |
+| | | to authenticate data transfer protocol. |
+| | | (See <<<dfs.data.transfer.protection>>>.) |
*-------------------------+-------------------------+------------------------+
| <<<dfs.datanode.http.address>>> | <0.0.0.0:1006> | |
| | | Secure DataNode must use privileged port |
@@ -457,6 +491,17 @@ Configuration for <<<conf/hdfs-site.xml>>>
| <<<dfs.encrypt.data.transfer>>> | <false> | |
| | | set to <<<true>>> when using data encryption |
*-------------------------+-------------------------+------------------------+
+| <<<dfs.data.transfer.protection>>> | | |
+| | | <authentication> : authentication only \
+| | | <integrity> : integrity check in addition to authentication \
+| | | <privacy> : data encryption in addition to integrity |
+| | | This property is unspecified by default. Setting this property enables |
+| | | SASL for authentication of data transfer protocol. If this is enabled, |
+| | | then <<<dfs.datanode.address>>> must use a non-privileged port, |
+| | | <<<dfs.http.policy>>> must be set to <HTTPS_ONLY> and the |
+| | | <<<HADOOP_SECURE_DN_USER>>> environment variable must be undefined when |
+| | | starting the DataNode process. |
+*-------------------------+-------------------------+------------------------+
Configuration for <<<conf/hdfs-site.xml>>>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/90971839/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 8fc4fc72..475d865 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -662,6 +662,9 @@ Release 2.6.0 - UNRELEASED
HDFS-7217. Better batching of IBRs. (kihwal)
+ HDFS-7195. Update user doc of secure mode about Datanodes don't require root
+ or jsvc. (cnauroth)
+
OPTIMIZATIONS
HDFS-6690. Deduplicate xattr names in memory. (wang)