You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2013/09/30 04:51:46 UTC
svn commit: r1527431 - in /incubator/knox: site/
site/books/knox-incubating-0-3-0/ trunk/books/0.3.0/
Author: kminder
Date: Mon Sep 30 02:51:46 2013
New Revision: 1527431
URL: http://svn.apache.org/r1527431
Log:
Add hostmap provider docs.
Modified:
incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
incubator/knox/site/index.html
incubator/knox/site/issue-tracking.html
incubator/knox/site/license.html
incubator/knox/site/mail-lists.html
incubator/knox/site/project-info.html
incubator/knox/site/team-list.html
incubator/knox/trunk/books/0.3.0/book_getting-started.md
incubator/knox/trunk/books/0.3.0/config.md
Modified: incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html (original)
+++ incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html Mon Sep 30 02:51:46 2013
@@ -189,7 +189,7 @@
<td><img src="question.png" alt="?"/> </td>
</tr>
<tr>
- <td>Hive/WebHCat </td>
+ <td>Hive (via WebHCat) </td>
<td>0.11.0 </td>
<td><img src="check.png" alt="y"/> </td>
<td><img src="check.png" alt="y"/> </td>
@@ -201,7 +201,7 @@
<td><img src="check.png" alt="y"/> </td>
</tr>
<tr>
- <td>Hive/JDBC </td>
+ <td>Hive (via JDBC) </td>
<td>0.11.0 </td>
<td><img src="error.png" alt="n"/> </td>
<td><img src="error.png" alt="n"/> </td>
@@ -213,7 +213,7 @@
<td><img src="question.png" alt="?"/> </td>
</tr>
<tr>
- <td>Hive/ODBC </td>
+ <td>Hive (via ODBC) </td>
<td>0.12.0 </td>
<td><img src="question.png" alt="?"/> </td>
<td><img src="question.png" alt="?"/> </td>
@@ -305,7 +305,60 @@ Server: Jetty(6.1.26)
</service>
</code></pre>
<dl><dt>/topology/service</dt><dd>Provider information about a particular service within the Hadoop cluster. Not all services are necessarily exposed as gateway endpoints.</dd><dt>/topology/service/role</dt><dd>Identifies the role of this service. Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER Additional service roles can be supported via plugins.</dd><dt>topology/service/url</dt><dd>The URL identifying the location of a particular service within the Hadoop cluster.</dd>
-</dl><h4><a id="Host+Mapping"></a>Host Mapping</h4><p>TODO - Complete Host Mapping docs.</p><p>That really depends upon how you have your VM configured. If you can hit <a href="http://c6401.ambari.apache.org:1022/">http://c6401.ambari.apache.org:1022/</a> directly from your client and knox host then you probably don’t need the hostmap at all. The host map only exists for situations where a host in the hadoop cluster is known by one name externally and another internally. For example running hostname -q on sandbox returns sandbox.hortonworks.com but externally Sandbox is setup to be accesses using localhost via portmapping. The way the hostmap config works is that the <name/> element is what the hadoop cluster host is known as externally and the <value/> is how the hadoop cluster host identifies itself internally. <param><name>localhost</name><value>c6401,c6401.ambari.apache.org</value></param> You SHOULD be able to simply change <enabled>true</enabled> to false but I have a su
spicion that that might not actually work. Please try it and file a jira if that doesn’t work. If so, simply either remove the full provider config for hostmap or remove the <param/> that defines the mapping.</p><h4><a id="Logging"></a>Logging</h4><p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the rootLogger value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><h4><a id="Java+VM+Options"></a>Java VM Options</h4><p>TODO - Java VM options doc.</p><h4><a id="Persisting+the+Master+Secret"></a>Persisting the Master Secret</h4><p>The master secret is required to start the server. This secret is used to access secured artifacts by the gateway instance. Keystore, trust stores and credential stores are all protected with the master secret.</p><p>You may persist the master s
ecret by supplying the <em>-persist-master</em> switch at startup. This will result in a warning indicating that persisting the secret is less secure than providing it at startup. We do make some provisions in order to protect the persisted password.</p><p>It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.</p><p>After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your environment. This is probably the most important layer of defense for master secret. Do not assume that the encryption if sufficient protection.</p><p>A specific user should be created to run the gateway this will protect a persisted master file.</p><h4><a id="Management+of+Security+Artifacts"></a>Management of Security Artifacts</h4><p>There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications,
access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
+</dl><h4><a id="Hostmap+Provider"></a>Hostmap Provider</h4><p>The purpose of the Hostmap provider is to handle situations where host are know by one name within the cluster and another name externally. This frequently occurs when virtual machines are used and in particular using cloud hosting services. Currently the Hostmap provider is configured as part of the topology file. The basic structure is shown below.</p>
+<pre><code><topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>external-host-name</name><value>internal-host-name</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+</topology>
+</code></pre><p>This mapping is required because the Hadoop servies running within the cluster are unaware that they are being accessed from outside the cluster. Therefore URLs returned as part of REST API responses will typically contain internal host names. Since clients outside the cluster will be unable to resolve those host name they must be mapped to external host names.</p><h5><a id="Hostmap+Provider+Example+-+EC2"></a>Hostmap Provider Example - EC2</h5><p>Consider an EC2 example where two VMs have been allocated. Each VM has an external host name by which it can be accessed via the internet. However the EC2 VM is unaware of this external host name and instead is configured with the internal host name.</p>
+<pre><code>External HOSTNAMES:
+ec2-23-22-31-165.compute-1.amazonaws.com
+ec2-23-23-25-10.compute-1.amazonaws.com
+
+Internal HOSTNAMES:
+ip-10-118-99-172.ec2.internal
+ip-10-39-107-209.ec2.internal
+</code></pre><p>The Hostmap configuration required to allow access external to the Hadoop cluster via the Apache Knox Gateway would be this.</p>
+<pre><code><topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>ec2-23-22-31-165.compute-1.amazonaws.com</name><value>ip-10-118-99-172.ec2.internal</value></param>
+ <param><name>ec2-23-23-25-10.compute-1.amazonaws.com</name><value>ip-10-39-107-209.ec2.internal</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+</topology>
+</code></pre><h5><a id="Hostmap+Provider+Example+-+Sandbox"></a>Hostmap Provider Example - Sandbox</h5><p>Hortonwork’s Sandbox 2.x poses a different challenge for host name mapping. This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it is accessible via localhost. However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the host name. So from the perspective of a client accessing Sandbox the external host name is localhost. The Hostmap configuration required to allow access to Sandbox from the host operating system is this.</p>
+<pre><code><topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+</topology>
+</code></pre><h5><a id="Hostmap+Provider+Configuration"></a>Hostmap Provider Configuration</h5><p>Details about each provider configuration element is enumerated below.</p>
+<dl><dt>topology/gateway/provider/role</dt><dd>The role for a Hostmap provider must always be <code>hostmap</code>.</dd><dt>topology/gateway/provider/name</dt><dd>The Hostmap provider supplied out-of-the-box is selected via the name <code>static</code>.</dd><dt>topology/gateway/provider/enabled</dt><dd>Host mapping can be enabled or disabled by providing <code>true</code> or <code>false</code>.</dd><dt>topology/gateway/provider/param</dt><dd>Host mapping is configured by providing parameters for each external to internal mapping.</dd><dt>topology/gateway/provider/param/name</dt><dd>The parameter names represent an external host names associated with the internal host names provided by the value element. This can be a comma separated list of host names that all represent the same physical host. When mapping from internal to external host name the first external host name in the list is used.</dd><dt>topology/gateway/provider/param/value</dt><dd>The parameter values represent the inte
rnal host names associated with the external host names provider by the name element. This can be a comma separated list of host names that all represent the same physical host. When mapping from external to internal host names the first internal host name in the list is used.</dd>
+</dl><h4><a id="Logging"></a>Logging</h4><p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the rootLogger value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><h4><a id="Java+VM+Options"></a>Java VM Options</h4><p>TODO - Java VM options doc.</p><h4><a id="Persisting+the+Master+Secret"></a>Persisting the Master Secret</h4><p>The master secret is required to start the server. This secret is used to access secured artifacts by the gateway instance. Keystore, trust stores and credential stores are all protected with the master secret.</p><p>You may persist the master secret by supplying the <em>-persist-master</em> switch at startup. This will result in a warning indicating that persisting the secret is less secure than providing it at startup. We do make some provisions in ord
er to protect the persisted password.</p><p>It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.</p><p>After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your environment. This is probably the most important layer of defense for master secret. Do not assume that the encryption if sufficient protection.</p><p>A specific user should be created to run the gateway this will protect a persisted master file.</p><h4><a id="Management+of+Security+Artifacts"></a>Management of Security Artifacts</h4><p>There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The followi
ng is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
<ol>
<li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
<ul>
@@ -347,7 +400,7 @@ Server: Jetty(6.1.26)
<li>All security related artifacts are protected with the master secret</li>
<li>Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways</li>
<li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
-</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h3><a id="Authentication"></a>Authentication</h3><p>TODO</p><h4><a id="LDAP+Configuration"></a>LDAP Configuration</h4><p>TODO</p><h5><a id="Creation+of+the+Key+Store+with+self+signed+certificate+and+enabling+it+on+Jetty"></a>Creation of the Key Store with self signed certificate and enabling it on Jetty</h5>
+</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as host name verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h3><a id="Authentication"></a>Authentication</h3><p>TODO</p><h4><a id="LDAP+Configuration"></a>LDAP Configuration</h4><p>TODO</p><h5><a id="Creation+of+the+Key+Store+with+self+signed+certificate+and+enabling+it+on+Jetty"></a>Creation of the Key Store with self signed certificate and enabling it on Jetty</h5>
<pre><code>keytool -keystore keystore -alias jetty -genkey -keyalg RSA -storepass secret
</code></pre><p>See more here about <a href="http://wiki.eclipse.org/Jetty/Howto/Configure_SSL">Jetty SSL setup</a></p><h5><a id="Shiro.ini+file+setup"></a>Shiro.ini file setup</h5><h6><a id="Shiro.ini"></a>Shiro.ini</h6>
<pre><code>[urls]
Modified: incubator/knox/site/index.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/index.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/index.html (original)
+++ incubator/knox/site/index.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/issue-tracking.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/issue-tracking.html (original)
+++ incubator/knox/site/issue-tracking.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/license.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/license.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/license.html (original)
+++ incubator/knox/site/license.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/mail-lists.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/mail-lists.html (original)
+++ incubator/knox/site/mail-lists.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/project-info.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/project-info.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/project-info.html (original)
+++ incubator/knox/site/project-info.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/team-list.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/team-list.html?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/site/team-list.html (original)
+++ incubator/knox/site/team-list.html Mon Sep 30 02:51:46 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 27, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Sep 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20130927" />
+ <meta name="Date-Revision-yyyymmdd" content="20130929" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-09-27</span>
+ | <span id="publishDate">Last Published: 2013-09-29</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/trunk/books/0.3.0/book_getting-started.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_getting-started.md?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_getting-started.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_getting-started.md Mon Sep 30 02:51:46 2013
@@ -151,17 +151,17 @@ The table below provides a brief explana
This table enumerates the versions of various Hadoop services that have been tested to work with the Knox Gateway.
Only more recent versions of some Hadoop components when secured via Kerberos can be accessed via the Knox Gateway.
-| Service | Version | Non-Secure | Secure |
-| ----------------- | ---------- | ----------- | ------ |
-| WebHDFS | 2.1.0 | ![y] | ![y] |
-| WebHCat/Templeton | 0.11.0 | ![y] | ![y] |
-| Ozzie | 4.0.0 | ![y] | ![y] |
-| HBase/Stargate | 0.95.2 | ![y] | ![?] |
-| Hive/WebHCat | 0.11.0 | ![y] | ![y] |
-| | 0.12.0 | ![y] | ![y] |
-| Hive/JDBC | 0.11.0 | ![n] | ![n] |
-| | 0.12.0 | ![?]![y] | ![?] |
-| Hive/ODBC | 0.12.0 | ![?] | ![?] |
+| Service | Version | Non-Secure | Secure |
+| ------------------ | ---------- | ----------- | ------ |
+| WebHDFS | 2.1.0 | ![y] | ![y] |
+| WebHCat/Templeton | 0.11.0 | ![y] | ![y] |
+| Ozzie | 4.0.0 | ![y] | ![y] |
+| HBase/Stargate | 0.95.2 | ![y] | ![?] |
+| Hive (via WebHCat) | 0.11.0 | ![y] | ![y] |
+| | 0.12.0 | ![y] | ![y] |
+| Hive (via JDBC) | 0.11.0 | ![n] | ![n] |
+| | 0.12.0 | ![?]![y] | ![?] |
+| Hive (via ODBC) | 0.12.0 | ![?] | ![?] |
### Basic Usage ###
Modified: incubator/knox/trunk/books/0.3.0/config.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/config.md?rev=1527431&r1=1527430&r2=1527431&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/config.md (original)
+++ incubator/knox/trunk/books/0.3.0/config.md Mon Sep 30 02:51:46 2013
@@ -112,19 +112,109 @@ Additional service roles can be supporte
topology/service/url
: The URL identifying the location of a particular service within the Hadoop cluster.
-#### Host Mapping ####
+#### Hostmap Provider ####
-TODO - Complete Host Mapping docs.
-
-That really depends upon how you have your VM configured.
-If you can hit http://c6401.ambari.apache.org:1022/ directly from your client and knox host then you probably don't need the hostmap at all.
-The host map only exists for situations where a host in the hadoop cluster is known by one name externally and another internally.
-For example running hostname -q on sandbox returns sandbox.hortonworks.com but externally Sandbox is setup to be accesses using localhost via portmapping.
-The way the hostmap config works is that the <name/> element is what the hadoop cluster host is known as externally and the <value/> is how the hadoop cluster host identifies itself internally.
-<param><name>localhost</name><value>c6401,c6401.ambari.apache.org</value></param>
-You SHOULD be able to simply change <enabled>true</enabled> to false but I have a suspicion that that might not actually work.
-Please try it and file a jira if that doesn't work.
-If so, simply either remove the full provider config for hostmap or remove the <param/> that defines the mapping.
+The purpose of the Hostmap provider is to handle situations where host are know by one name within the cluster and another name externally.
+This frequently occurs when virtual machines are used and in particular using cloud hosting services.
+Currently the Hostmap provider is configured as part of the topology file.
+The basic structure is shown below.
+
+ <topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>external-host-name</name><value>internal-host-name</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+ </topology>
+
+This mapping is required because the Hadoop servies running within the cluster are unaware that they are being accessed from outside the cluster.
+Therefore URLs returned as part of REST API responses will typically contain internal host names.
+Since clients outside the cluster will be unable to resolve those host name they must be mapped to external host names.
+
+##### Hostmap Provider Example - EC2 #####
+
+Consider an EC2 example where two VMs have been allocated.
+Each VM has an external host name by which it can be accessed via the internet.
+However the EC2 VM is unaware of this external host name and instead is configured with the internal host name.
+
+ External HOSTNAMES:
+ ec2-23-22-31-165.compute-1.amazonaws.com
+ ec2-23-23-25-10.compute-1.amazonaws.com
+
+ Internal HOSTNAMES:
+ ip-10-118-99-172.ec2.internal
+ ip-10-39-107-209.ec2.internal
+
+The Hostmap configuration required to allow access external to the Hadoop cluster via the Apache Knox Gateway would be this.
+
+ <topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>ec2-23-22-31-165.compute-1.amazonaws.com</name><value>ip-10-118-99-172.ec2.internal</value></param>
+ <param><name>ec2-23-23-25-10.compute-1.amazonaws.com</name><value>ip-10-39-107-209.ec2.internal</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+ </topology>
+
+##### Hostmap Provider Example - Sandbox #####
+
+Hortonwork's Sandbox 2.x poses a different challenge for host name mapping.
+This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it is accessible via localhost.
+However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the host name.
+So from the perspective of a client accessing Sandbox the external host name is localhost.
+The Hostmap configuration required to allow access to Sandbox from the host operating system is this.
+
+ <topology>
+ <gateway>
+ ...
+ <provider>
+ <role>hostmap</role>
+ <name>static</name>
+ <enabled>true</enabled>
+ <param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param>
+ </provider>
+ ...
+ </gateway>
+ ...
+ </topology>
+
+##### Hostmap Provider Configuration #####
+
+Details about each provider configuration element is enumerated below.
+
+topology/gateway/provider/role
+: The role for a Hostmap provider must always be `hostmap`.
+
+topology/gateway/provider/name
+: The Hostmap provider supplied out-of-the-box is selected via the name `static`.
+
+topology/gateway/provider/enabled
+: Host mapping can be enabled or disabled by providing `true` or `false`.
+
+topology/gateway/provider/param
+: Host mapping is configured by providing parameters for each external to internal mapping.
+
+topology/gateway/provider/param/name
+: The parameter names represent an external host names associated with the internal host names provided by the value element.
+This can be a comma separated list of host names that all represent the same physical host.
+When mapping from internal to external host name the first external host name in the list is used.
+
+topology/gateway/provider/param/value
+: The parameter values represent the internal host names associated with the external host names provider by the name element.
+This can be a comma separated list of host names that all represent the same physical host.
+When mapping from external to internal host names the first internal host name in the list is used.
#### Logging ####
@@ -248,6 +338,6 @@ Once you have created these keystores yo
This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.
NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster.
-When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue.
+When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as host name verification problems will ensue.
Obviously, trust-stores will need to be taken into account as well.