You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by lm...@apache.org on 2013/03/16 15:39:11 UTC
[2/2] git commit: Updated to be in sync with readme.md
Updated Branches:
refs/heads/master 282261ff6 -> ee4df3759
Updated to be in sync with readme.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-knox/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-knox/commit/ee4df375
Tree: http://git-wip-us.apache.org/repos/asf/incubator-knox/tree/ee4df375
Diff: http://git-wip-us.apache.org/repos/asf/incubator-knox/diff/ee4df375
Branch: refs/heads/master
Commit: ee4df37597b862383eb21a92d187ed13ba93aeea
Parents: b7f3cb4
Author: Larry McCay <lm...@hortonworks.com>
Authored: Sat Mar 16 10:38:54 2013 -0400
Committer: Larry McCay <lm...@hortonworks.com>
Committed: Sat Mar 16 10:38:54 2013 -0400
----------------------------------------------------------------------
gateway-release/README | 94 +++++++++++++++++++++++++++++++++++++++++-
1 files changed, 91 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-knox/blob/ee4df375/gateway-release/README
----------------------------------------------------------------------
diff --git a/gateway-release/README b/gateway-release/README
index ccced99..7eb06a5 100644
--- a/gateway-release/README
+++ b/gateway-release/README
@@ -129,9 +129,15 @@ Installation and Deployment Instructions
can safely be ignored.
4. Start the Gateway server
- java -jar bin/gateway-0.2.0-SNAPSHOT.jar
- Take note of the port identified in the log output as you will need this
- for accessing the Gateway.
+ java -jar bin/gateway-server-0.2.0-SNAPSHOT.jar
+ a. Take note of the port identified in the logging output as you will need this for
+ accessing the gateway.
+ b. The server will prompt you for the master secret (password). This secret is used
+ to secure artifacts used to secure artifacts used by the gateway server for
+ things like SSL, credential/password aliasing. This secret will have to be entered
+ at startup unless you choose to persist it. Remember this secret and keep it safe.
+ It represents the keys to the kingdom. See the Persisting the Master section for
+ more information.
5. Configure the Gateway with the topology of your Hadoop cluster
a. Edit the file {GATEWAY_HOME}/deployments/sample.xml
@@ -183,6 +189,88 @@ Installation and Deployment Instructions
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html
http://people.apache.org/~thejas/templeton_doc_v1/
http://oozie.apache.org/docs/3.3.1/WebServicesAPI.html
+
+------------------------------------------------------------------------------
+Persisting the Master
+------------------------------------------------------------------------------
+The master secret is required to start the server. This secret is used to access secured artifacts by the gateway
+instance. Keystore, trust stores and credential stores are all protected with the master secret.
+
+You may persist the master secret by supplying the *-persist-master* switch at startup. This will result in a
+warning indicating that persisting the secret is less secure than providing it at startup. We do make some provisions in
+order to protect the persisted password.
+
+It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessable by
+the user that the gateway is running as.
+
+After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your
+environment. This is probably the most important layer of defense for master secret. Do not assume that the encryption if
+sufficient protection.
+
+A specific user should be created to run the gateway this will protect a persisted master file.
+
+------------------------------------------------------------------------------
+Management of Security Artifacts
+------------------------------------------------------------------------------
+There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications,
+access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of
+the gateway instances or generated and populated by the gateway instance itself.
+
+The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway
+instances and instances as part of a cluster of gateways in mind.
+
+Upon start of the gateway server we:
+
+1. Look for an identity store at conf/security/keystores/gateway.jks. The identity store contains the certificate
+ and private key used to represent the identity of the server for SSL connections and signtature creation.
+ a. If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo
+ mode. The certificate is stored with an alias of gateway-identity.
+ b. If there is an identity store found than we ensure that it can be loaded using the provided master secret and
+ that there is an alias with called gateway-identity.
+2. Look for a credential store at conf/security/keystores/__gateway-credentials.jceks. This credential store is used
+ to store secrets/passwords that are used by the gateway. For instance, this is where the passphrase for accessing
+ the gateway-identity certificate is kept.
+ a. If there is no credential store found then we create one and populate it with a generated passphrase for the alias
+ gateway-identity-passphrase. This is coordinated with the population of the self-signed cert into the identity-store.
+ b. If a credential store is found then we ensure that it can be loaded using the provided master secret and that the
+ expected aliases have been populated with secrets.
+
+Upon deployment of a Hadoop cluster topology within the gateway we:
+
+1. Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box.
+ We look for conf/security/keystores/sample-credentials.jceks. This topology specific credential store is used for storing
+ secrets/passwords that are used for encrypting sensitive data with topology specific keys.
+ a. If no credential store is found for the topology being deployed then one is created for it. Population of the aliases
+ is delegated to the configured providers within the system that will require the use of a secret for a particular
+ task. They may programmatically set the value of the secret or choose to have the value for the specified alias
+ generated through the AliasService..
+ b. If a credential store is found then we ensure that it can be loaded with the provided master secret and the confgured
+ providers have the opportunity to ensure that the aliases are populated and if not to populate them.
+
+ By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a
+ number of ways.
+
+ 1. Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location
+ and then replicated across all of the slave instances before startup.
+ 2. Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to
+ replicate them over the network. Of course, NFS mounts have their own challenges.
+
+Summary of Secrets to be Managed:
+
+1. Master secret - the same for all gateway instances in a cluster of gateways
+2. All security related artifacts are protected with the master secret
+3. Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway
+ instances in the cluster of gateways
+4. Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same
+ for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets
+ for one hadoop cluster are different from those of another. This allows for failover from one gateway instance to another
+ even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.
+
+NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able
+to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity
+store will not be able to be blindly replicated as hostname verification problems will ensue. Obviously, truststores will
+need to be taken into account as well.
+
------------------------------------------------------------------------------
Mapping Gateway URLs to Hadoop cluster URLs