You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by lm...@apache.org on 2014/09/24 16:18:29 UTC
svn commit: r1627328 - in /knox: site/ site/books/knox-0-4-0/
site/books/knox-0-5-0/ trunk/books/0.5.0/
Author: lmccay
Date: Wed Sep 24 14:18:28 2014
New Revision: 1627328
URL: http://svn.apache.org/r1627328
Log:
KNOX-429 updates to user guide for 0.5.0 via Sumit Gupta
Modified:
knox/site/books/knox-0-4-0/deployment-overview.png
knox/site/books/knox-0-4-0/deployment-provider.png
knox/site/books/knox-0-4-0/deployment-service.png
knox/site/books/knox-0-4-0/runtime-overview.png
knox/site/books/knox-0-4-0/runtime-request-processing.png
knox/site/books/knox-0-5-0/knox-0-5-0.html
knox/site/index.html
knox/site/issue-tracking.html
knox/site/license.html
knox/site/mail-lists.html
knox/site/project-info.html
knox/site/team-list.html
knox/trunk/books/0.5.0/book_getting-started.md
knox/trunk/books/0.5.0/config.md
knox/trunk/books/0.5.0/config_authn.md
knox/trunk/books/0.5.0/quick_start.md
Modified: knox/site/books/knox-0-4-0/deployment-overview.png
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-4-0/deployment-overview.png?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
Binary files - no diff available.
Modified: knox/site/books/knox-0-4-0/deployment-provider.png
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-4-0/deployment-provider.png?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
Binary files - no diff available.
Modified: knox/site/books/knox-0-4-0/deployment-service.png
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-4-0/deployment-service.png?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
Binary files - no diff available.
Modified: knox/site/books/knox-0-4-0/runtime-overview.png
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-4-0/runtime-overview.png?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
Binary files - no diff available.
Modified: knox/site/books/knox-0-4-0/runtime-request-processing.png
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-4-0/runtime-request-processing.png?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
Binary files - no diff available.
Modified: knox/site/books/knox-0-5-0/knox-0-5-0.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-5-0/knox-0-5-0.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/books/knox-0-5-0/knox-0-5-0.html (original)
+++ knox/site/books/knox-0-5-0/knox-0-5-0.html Wed Sep 24 14:18:28 2014
@@ -76,20 +76,20 @@
<li>Do Hadoop with Knox</li>
</ol><h3><a id="1+-+Requirements"></a>1 - Requirements</h3><h4><a id="Java"></a>Java</h4><p>Java 1.6 or later is required for the Knox Gateway runtime. Use the command below to check the version of Java installed on the system where Knox will be running.</p>
<pre><code>java -version
-</code></pre><h4><a id="Hadoop"></a>Hadoop</h4><p>Knox 0.4.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment. </p><h3><a id="2+-+Download+Hadoop+2.x+VM"></a>2 - Download Hadoop 2.x VM</h3><p>The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine <a href="http://hortonworks.com/products/hdp-2/#install">Sandbox</a>. Please note Knox supports other Hadoop distributions and is configurable against a full blown Hadoop cluster. Configuring Knox for Hadoop 2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster is documented in advance deployment guide.</p><h3><a id="3+-+Download+Apache+Knox+Gateway"></a>3 - Download Apache Knox Gateway</h3><p>Download one of the distributions below from the <a href="http://www.apache.org/dyn/closer.cgi/knox">Apache mirrors</a>.</p>
+</code></pre><h4><a id="Hadoop"></a>Hadoop</h4><p>Knox 0.5.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment. </p><h3><a id="2+-+Download+Hadoop+2.x+VM"></a>2 - Download Hadoop 2.x VM</h3><p>The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine <a href="http://hortonworks.com/products/hdp-2/#install">Sandbox</a>. Please note Knox supports other Hadoop distributions and is configurable against a full blown Hadoop cluster. Configuring Knox for Hadoop 2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster is documented in advance deployment guide.</p><h3><a id="3+-+Download+Apache+Knox+Gateway"></a>3 - Download Apache Knox Gateway</h3><p>Download one of the distributions below from the <a href="http://www.apache.org/dyn/closer.cgi/knox">Apache mirrors</a>.</p>
<ul>
- <li>Source archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip">knox-0.4.0-src.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.md5">MD5 digest</a>)</li>
- <li>Binary archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip">knox-0.4.0.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.md5">MD5 digest</a>)</li>
+ <li>Source archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.4.0-src.zip">knox-0.5.0-src.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.md5">MD5 digest</a>)</li>
+ <li>Binary archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip">knox-0.5.0.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.md5">MD5 digest</a>)</li>
</ul><p>Apache Knox Gateway releases are available under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>. See the NOTICE file contained in each release artifact for applicable copyright attribution notices.</p><h3><a id="Verify"></a>Verify</h3><p>While recommended, verify is an optional step. You can verify the integrity of any downloaded files using the PGP signatures. Please read <a href="http://httpd.apache.org/dev/verification.html">Verifying Apache HTTP Server Releases</a> for more information on why you should verify our releases.</p><p>The PGP signatures can be verified using PGP or GPG. First download the KEYS file as well as the .asc signature files for the relevant release packages. Make sure you get these files from the main distribution directory linked above, rather than from a mirror. Then verify the signatures using one of the methods below.</p>
<pre><code>% pgpk -a KEYS
-% pgpv knox-0.4.0.zip.asc
+% pgpv knox-0.5.0.zip.asc
</code></pre><p>or</p>
<pre><code>% pgp -ka KEYS
-% pgp knox-0.4.0.zip.asc
+% pgp knox-0.5.0.zip.asc
</code></pre><p>or</p>
<pre><code>% gpg --import KEYS
-% gpg --verify knox-0.4.0.zip.asc
-</code></pre><h3><a id="4+-+Start+Hadoop+virtual+machine"></a>4 - Start Hadoop virtual machine</h3><p>Start the Hadoop virtual machine.</p><h3><a id="5+-+Install+Knox"></a>5 - Install Knox</h3><p>The steps required to install the gateway will vary depending upon which distribution format (zip | rpm) was downloaded. In either case you will end up with a directory where the gateway is installed. This directory will be referred to as your <code>{GATEWAY_HOME}</code> throughout this document.</p><h4><a id="ZIP"></a>ZIP</h4><p>If you downloaded the Zip distribution you can simply extract the contents into a directory. The example below provides a command that can be executed to do this. Note the <code>{VERSION}</code> portion of the command must be replaced with an actual Apache Knox Gateway version number. This might be 0.4.0 for example and must patch the value in the file downloaded.</p>
+% gpg --verify knox-0.5.0.zip.asc
+</code></pre><h3><a id="5+-+Start+Hadoop+virtual+machine"></a>5 - Start Hadoop virtual machine</h3><p>Start the Hadoop virtual machine.</p><h3><a id="5+-+Install+Knox"></a>5 - Install Knox</h3><p>The steps required to install the gateway will vary depending upon which distribution format (zip | rpm) was downloaded. In either case you will end up with a directory where the gateway is installed. This directory will be referred to as your <code>{GATEWAY_HOME}</code> throughout this document.</p><h4><a id="ZIP"></a>ZIP</h4><p>If you downloaded the Zip distribution you can simply extract the contents into a directory. The example below provides a command that can be executed to do this. Note the <code>{VERSION}</code> portion of the command must be replaced with an actual Apache Knox Gateway version number. This might be 0.4.0 for example and must patch the value in the file downloaded.</p>
<pre><code>jar xf knox-{VERSION}.zip
</code></pre><p>This will create a directory <code>knox-{VERSION}</code> in your current directory. The directory <code>knox-{VERSION}</code> will considered your <code>{GATEWAY_HOME}</code></p><h3><a id="6+-+Start+LDAP+embedded+in+Knox"></a>6 - Start LDAP embedded in Knox</h3><p>Knox comes with an LDAP server for demonstration purposes.</p>
<pre><code>cd {GATEWAY_HOME}
@@ -121,7 +121,7 @@ Server: Jetty(6.1.26)
{"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350596040075,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350595857178,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"type":"DIRECTORY"}
]}}
-</code></pre><h4><a id="Submit+a+MR+job+via+Knox."></a>Submit a MR job via Knox.</h4><h4><a id="Get+status+of+a+MR+job+via+Knox."></a>Get status of a MR job via Knox.</h4><h4><a id="Cancel+a+MR+job+via+Knox."></a>Cancel a MR job via Knox.</h4><h3><a id="More+Examples"></a>More Examples</h3><h2><a id="Apache+Knox+Details"></a>Apache Knox Details</h2><p>This section provides everything you need to know to get the Knox gateway up and running against a Hadoop cluster.</p><h4><a id="Hadoop"></a>Hadoop</h4><p>An existing Hadoop 2.x cluster is required for Knox 0.4.0 to sit in front of and protect. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here. It is also possible to protect access to a services of a Hadoop cluster that is secured with kerberos. This too requires additional configuration that is described in other sections of this guide. See <a href="#Supported+Services">Supported Services</a> for details on what is s
upported for this release.</p><p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow assume a few things:</p>
+</code></pre><h4><a id="Submit+a+MR+job+via+Knox."></a>Submit a MR job via Knox.</h4><h4><a id="Get+status+of+a+MR+job+via+Knox."></a>Get status of a MR job via Knox.</h4><h4><a id="Cancel+a+MR+job+via+Knox."></a>Cancel a MR job via Knox.</h4><h3><a id="More+Examples"></a>More Examples</h3><h2><a id="Apache+Knox+Details"></a>Apache Knox Details</h2><p>This section provides everything you need to know to get the Knox gateway up and running against a Hadoop cluster.</p><h4><a id="Hadoop"></a>Hadoop</h4><p>An existing Hadoop 2.x cluster is required for Knox 0.5.0 to sit in front of and protect. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here. It is also possible to protect access to a services of a Hadoop cluster that is secured with kerberos. This too requires additional configuration that is described in other sections of this guide. See <a href="#Supported+Services">Supported Services</a> for details on what is s
upported for this release.</p><p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow assume a few things:</p>
<ol>
<li>The gateway is <em>not</em> collocated with the Hadoop clusters themselves.</li>
<li>The host names and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running.</li>
@@ -264,6 +264,12 @@ Server: Jetty(6.1.26)
<td><img src="check.png" alt="y"/> </td>
<td><img src="check.png" alt="y"/> </td>
</tr>
+ <tr>
+ <td>Yarn </td>
+ <td>2.5.0 </td>
+ <td><img src="check.png" alt="y"/> </td>
+ <td><img src="check.png" alt="y"/> </td>
+ </tr>
</tbody>
</table><h3><a id="More+Examples"></a>More Examples</h3><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
<ul>
@@ -272,6 +278,7 @@ Server: Jetty(6.1.26)
<li><a href="#Oozie+Examples">Oozie Examples</a></li>
<li><a href="#HBase+Examples">HBase Examples</a></li>
<li><a href="#Hive+Examples">Hive Examples</a></li>
+ <li><a href="#Yarn+Examples">Yarn Examples</a></li>
</ul><h2><a id="Gateway+Details"></a>Gateway Details</h2><p>This section describes the details of the Knox Gateway itself. Including: </p>
<ul>
<li>How URLs are mapped between a gateway that services multiple Hadoop clusters and the clusters themselves</li>
@@ -336,7 +343,7 @@ https://{gateway-host}:{gateway-port}/{g
<url>http://localhost:50070/webhdfs</url>
</service>
</code></pre>
-<dl><dt>/topology/service</dt><dd>Provider information about a particular service within the Hadoop cluster. Not all services are necessarily exposed as gateway endpoints.</dd><dt>/topology/service/role</dt><dd>Identifies the role of this service. Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER Additional service roles can be supported via plugins.</dd><dt>topology/service/url</dt><dd>The URL identifying the location of a particular service within the Hadoop cluster.</dd>
+<dl><dt>/topology/service</dt><dd>Provider information about a particular service within the Hadoop cluster. Not all services are necessarily exposed as gateway endpoints.</dd><dt>/topology/service/role</dt><dd>Identifies the role of this service. Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER, RESOURCEMANAGER Additional service roles can be supported via plugins.</dd><dt>topology/service/url</dt><dd>The URL identifying the location of a particular service within the Hadoop cluster.</dd>
</dl><h4><a id="Hostmap+Provider"></a>Hostmap Provider</h4><p>The purpose of the Hostmap provider is to handle situations where host are known by one name within the cluster and another name externally. This frequently occurs when virtual machines are used and in particular when using cloud hosting services. Currently, the Hostmap provider is configured as part of the topology file. The basic structure is shown below.</p>
<pre><code><topology>
<gateway>
@@ -615,7 +622,7 @@ ldapRealm=org.apache.shiro.realm.ldap.Jn
ldapRealm.contextFactory.authenticationMechanism=simple
ldapRealm.contextFactory.url=ldap://localhost:33389
ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
-</code></pre><p>In order to fit into the context of an INI file format, at deployment time we interrogate the paramaters provided in the provider configuration and parse the INI section out of the paramter names. The following provider config illustrates this approach. Notice that the section names in the above shiro.ini match the beginning of the param names that are in the following config:</p>
+</code></pre><p>In order to fit into the context of an INI file format, at deployment time we interrogate the parameters provided in the provider configuration and parse the INI section out of the parameter names. The following provider config illustrates this approach. Notice that the section names in the above shiro.ini match the beginning of the param names that are in the following config:</p>
<pre><code><gateway>
<provider>
<role>authentication</role>
@@ -642,7 +649,7 @@ ldapRealm.userDnTemplate=uid={0},ou=peop
<value>authcBasic</value>
</param>
</provider>
-</code></pre><p>This happens to be the way that we are currently configuring Shiro for BASIC/LDAP authentication. This same config approach may be used to achieve other authentication mechanisms or variations on this one. We however have not tested additional uses for it for this release.</p><h4><a id="LDAP+Configuration"></a>LDAP Configuration</h4><p>This section discusses the LDAP configuration used above for the Shiro Provider. Some of these configuration elements will need to be customized to reflect your deployment environment.</p><p><strong>main.ldapRealm</strong> - this element indicates the fully qualified classname of the Shiro realm to be used in authenticating the user. The classname provided by default in the sample is the <code>org.apache.shiro.realm.ldap.JndiLdapRealm</code> this implementation provides us with the ability to authenticate but by default has authorization disabled. In order to provide authorization - which is seen by Shiro as dependent on an LDAP schema
that is specific to each organization - an extension of JndiLdapRealm is generally used to override and implement the doGetAuhtorizationInfo method. In this particular release we are providing a simple authorization provider that can be used along with the Shiro authentication provider.</p><p><strong>main.ldapRealm.userDnTemplate</strong> - in order to bind a simple username to an LDAP server that generally requires a full distinguished name (DN), we must provide the template into which the simple username will be inserted. This template allows for the creation of a DN by injecting the simple username into the common name (CN) portion of the DN. <strong>This element will need to be customized to reflect your deployment environment.</strong> The template provided in the sample is only an example and is valid only within the LDAP schema distributed with Knox and is represented by the users.ldif file in the {GATEWAY_HOME}conf directory.</p><p><strong>main.ldapRealm.contextFactory.url<
/strong> - this element is the URL that represents the host and port of LDAP server. It also includes the scheme of the protocol to use. This may be either ldap or ldaps depending on whether you are communicating with the LDAP over SSL (higly recommended). <strong>This element will need to be cusomized to reflect your deployment environment.</strong>.</p><p><strong>main.ldapRealm.contextFactory.authenticationMechanism</strong> - this element indicates the type of authentication that should be performed against the LDAP server. The current default value is <code>simple</code> which indicates a simple bind operation. This element should not need to be modified and no mechanism other than a simple bind has been tested for this particular release.</p><p><strong>urls./</strong>** - this element represents a single URL_Ant_Path_Expression and the value the Shiro filter chain to apply to it. This particular sample indicates that all paths into the application have the same Shiro filter cha
in applied. The paths are relative to the application context path. The use of the value <code>authcBasic</code> here indicates that BASIC authentication is expected for every path into the application. Adding an additional Shiro filter to that chain for validating that the request isSecure() and over SSL can be achieved by changing the value to <code>ssl, authcBasic</code>. It is not likely that you need to change this element for your environment.</p><h4><a id="Active+Directory+-+Special+Note"></a>Active Directory - Special Note</h4><p>You would use LDAP configuration as documented above to authenticate against Active Directory as well.</p><p>Some Active Directory specifc things to keep in mind:</p><p>Typical AD main.ldapRealm.userDnTemplate value looks slightly different, such as cn={0},cn=users,DC=lab,DC=sample,dc=com</p><p>Please compare this with a typical Apache DS main.ldapRealm.userDnTemplate value and make note of the difference. uid={0},ou=people,dc=hadoop,dc=apache,dc=
org</p><p>If your AD is configured to authenticate based on just the cn and password and does not require user DN, you do not have to specify value for main.ldapRealm.userDnTemplate.</p><h4><a id="LDAP+over+SSL+(LDAPS)+Configuration"></a>LDAP over SSL (LDAPS) Configuration</h4><p>In order to communicate with your LDAP server over SSL (again, highly recommended), you will need to modify the topology file in a couple ways and possibly provision some keying material.</p>
+</code></pre><p>This happens to be the way that we are currently configuring Shiro for BASIC/LDAP authentication. This same config approach may be used to achieve other authentication mechanisms or variations on this one. We however have not tested additional uses for it for this release.</p><h4><a id="LDAP+Configuration"></a>LDAP Configuration</h4><p>This section discusses the LDAP configuration used above for the Shiro Provider. Some of these configuration elements will need to be customized to reflect your deployment environment.</p><p><strong>main.ldapRealm</strong> - this element indicates the fully qualified classname of the Shiro realm to be used in authenticating the user. The classname provided by default in the sample is the <code>org.apache.shiro.realm.ldap.JndiLdapRealm</code> this implementation provides us with the ability to authenticate but by default has authorization disabled. In order to provide authorization - which is seen by Shiro as dependent on an LDAP schema
that is specific to each organization - an extension of JndiLdapRealm is generally used to override and implement the doGetAuhtorizationInfo method. In this particular release we are providing a simple authorization provider that can be used along with the Shiro authentication provider.</p><p><strong>main.ldapRealm.userDnTemplate</strong> - in order to bind a simple username to an LDAP server that generally requires a full distinguished name (DN), we must provide the template into which the simple username will be inserted. This template allows for the creation of a DN by injecting the simple username into the common name (CN) portion of the DN. <strong>This element will need to be customized to reflect your deployment environment.</strong> The template provided in the sample is only an example and is valid only within the LDAP schema distributed with Knox and is represented by the users.ldif file in the {GATEWAY_HOME}/conf directory.</p><p><strong>main.ldapRealm.contextFactory.url
</strong> - this element is the URL that represents the host and port of LDAP server. It also includes the scheme of the protocol to use. This may be either ldap or ldaps depending on whether you are communicating with the LDAP over SSL (higly recommended). <strong>This element will need to be cusomized to reflect your deployment environment.</strong>.</p><p><strong>main.ldapRealm.contextFactory.authenticationMechanism</strong> - this element indicates the type of authentication that should be performed against the LDAP server. The current default value is <code>simple</code> which indicates a simple bind operation. This element should not need to be modified and no mechanism other than a simple bind has been tested for this particular release.</p><p><strong>urls./</strong>** - this element represents a single URL_Ant_Path_Expression and the value the Shiro filter chain to apply to it. This particular sample indicates that all paths into the application have the same Shiro filter ch
ain applied. The paths are relative to the application context path. The use of the value <code>authcBasic</code> here indicates that BASIC authentication is expected for every path into the application. Adding an additional Shiro filter to that chain for validating that the request isSecure() and over SSL can be achieved by changing the value to <code>ssl, authcBasic</code>. It is not likely that you need to change this element for your environment.</p><h4><a id="Active+Directory+-+Special+Note"></a>Active Directory - Special Note</h4><p>You would use LDAP configuration as documented above to authenticate against Active Directory as well.</p><p>Some Active Directory specifc things to keep in mind:</p><p>Typical AD main.ldapRealm.userDnTemplate value looks slightly different, such as cn={0},cn=users,DC=lab,DC=sample,dc=com</p><p>Please compare this with a typical Apache DS main.ldapRealm.userDnTemplate value and make note of the difference. uid={0},ou=people,dc=hadoop,dc=apache,dc
=org</p><p>If your AD is configured to authenticate based on just the cn and password and does not require user DN, you do not have to specify value for main.ldapRealm.userDnTemplate.</p><h4><a id="LDAP+over+SSL+(LDAPS)+Configuration"></a>LDAP over SSL (LDAPS) Configuration</h4><p>In order to communicate with your LDAP server over SSL (again, highly recommended), you will need to modify the topology file in a couple ways and possibly provision some keying material.</p>
<ol>
<li><strong>main.ldapRealm.contextFactory.url</strong> must be changed to have the <code>ldaps</code> protocol scheme and the port must be the SSL listener port on your LDAP server.</li>
<li>Identity certificate (keypair) provisioned to LDAP server - your LDAP server specific documentation should indicate what is requried for providing a cert or keypair to represent the LDAP server identity to connecting clients.</li>
Modified: knox/site/index.html
URL: http://svn.apache.org/viewvc/knox/site/index.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/index.html (original)
+++ knox/site/index.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/knox/site/issue-tracking.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/issue-tracking.html (original)
+++ knox/site/issue-tracking.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/license.html
URL: http://svn.apache.org/viewvc/knox/site/license.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/license.html (original)
+++ knox/site/license.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/knox/site/mail-lists.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/mail-lists.html (original)
+++ knox/site/mail-lists.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/project-info.html
URL: http://svn.apache.org/viewvc/knox/site/project-info.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/project-info.html (original)
+++ knox/site/project-info.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/team-list.html
URL: http://svn.apache.org/viewvc/knox/site/team-list.html?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/site/team-list.html (original)
+++ knox/site/team-list.html Wed Sep 24 14:18:28 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-17 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.6 at 2014-09-24 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140917" />
+ <meta name="Date-Revision-yyyymmdd" content="20140924" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-09-17</span>
+ | <span id="publishDate">Last Published: 2014-09-24</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/trunk/books/0.5.0/book_getting-started.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_getting-started.md?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_getting-started.md (original)
+++ knox/trunk/books/0.5.0/book_getting-started.md Wed Sep 24 14:18:28 2014
@@ -21,7 +21,7 @@ This section provides everything you nee
#### Hadoop ####
-An existing Hadoop 2.x cluster is required for Knox 0.4.0 to sit in front of and protect.
+An existing Hadoop 2.x cluster is required for Knox 0.5.0 to sit in front of and protect.
It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here.
It is also possible to protect access to a services of a Hadoop cluster that is secured with kerberos.
This too requires additional configuration that is described in other sections of this guide.
@@ -80,6 +80,7 @@ This table enumerates the versions of va
| HBase/Stargate | 0.98.0 | ![y] | ![y] |
| Hive (via WebHCat) | 0.13.0 | ![y] | ![y] |
| Hive (via JDBC) | 0.13.0 | ![y] | ![y] |
+| Yarn | 2.5.0 | ![y] | ![y] |
### More Examples ###
@@ -91,3 +92,4 @@ These examples provide more detail about
* #[Oozie Examples]
* #[HBase Examples]
* #[Hive Examples]
+* #[Yarn Examples]
Modified: knox/trunk/books/0.5.0/config.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config.md?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/config.md (original)
+++ knox/trunk/books/0.5.0/config.md Wed Sep 24 14:18:28 2014
@@ -106,7 +106,7 @@ Not all services are necessarily exposed
/topology/service/role
: Identifies the role of this service.
-Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER
+Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER, RESOURCEMANAGER
Additional service roles can be supported via plugins.
topology/service/url
Modified: knox/trunk/books/0.5.0/config_authn.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_authn.md?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/config_authn.md (original)
+++ knox/trunk/books/0.5.0/config_authn.md Wed Sep 24 14:18:28 2014
@@ -61,7 +61,7 @@ The following example illustrates a conf
ldapRealm.contextFactory.url=ldap://localhost:33389
ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
-In order to fit into the context of an INI file format, at deployment time we interrogate the paramaters provided in the provider configuration and parse the INI section out of the paramter names. The following provider config illustrates this approach. Notice that the section names in the above shiro.ini match the beginning of the param names that are in the following config:
+In order to fit into the context of an INI file format, at deployment time we interrogate the parameters provided in the provider configuration and parse the INI section out of the parameter names. The following provider config illustrates this approach. Notice that the section names in the above shiro.ini match the beginning of the param names that are in the following config:
<gateway>
<provider>
@@ -98,7 +98,7 @@ This section discusses the LDAP configur
**main.ldapRealm** - this element indicates the fully qualified classname of the Shiro realm to be used in authenticating the user. The classname provided by default in the sample is the `org.apache.shiro.realm.ldap.JndiLdapRealm` this implementation provides us with the ability to authenticate but by default has authorization disabled. In order to provide authorization - which is seen by Shiro as dependent on an LDAP schema that is specific to each organization - an extension of JndiLdapRealm is generally used to override and implement the doGetAuhtorizationInfo method. In this particular release we are providing a simple authorization provider that can be used along with the Shiro authentication provider.
-**main.ldapRealm.userDnTemplate** - in order to bind a simple username to an LDAP server that generally requires a full distinguished name (DN), we must provide the template into which the simple username will be inserted. This template allows for the creation of a DN by injecting the simple username into the common name (CN) portion of the DN. **This element will need to be customized to reflect your deployment environment.** The template provided in the sample is only an example and is valid only within the LDAP schema distributed with Knox and is represented by the users.ldif file in the {GATEWAY_HOME}conf directory.
+**main.ldapRealm.userDnTemplate** - in order to bind a simple username to an LDAP server that generally requires a full distinguished name (DN), we must provide the template into which the simple username will be inserted. This template allows for the creation of a DN by injecting the simple username into the common name (CN) portion of the DN. **This element will need to be customized to reflect your deployment environment.** The template provided in the sample is only an example and is valid only within the LDAP schema distributed with Knox and is represented by the users.ldif file in the {GATEWAY_HOME}/conf directory.
**main.ldapRealm.contextFactory.url** - this element is the URL that represents the host and port of LDAP server. It also includes the scheme of the protocol to use. This may be either ldap or ldaps depending on whether you are communicating with the LDAP over SSL (higly recommended). **This element will need to be cusomized to reflect your deployment environment.**.
Modified: knox/trunk/books/0.5.0/quick_start.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/quick_start.md?rev=1627328&r1=1627327&r2=1627328&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/quick_start.md (original)
+++ knox/trunk/books/0.5.0/quick_start.md Wed Sep 24 14:18:28 2014
@@ -41,7 +41,7 @@ Use the command below to check the versi
#### Hadoop ####
-Knox 0.4.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment.
+Knox 0.5.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment.
### 2 - Download Hadoop 2.x VM ###
@@ -53,17 +53,17 @@ Configuring Knox for Hadoop 2.x version,
Download one of the distributions below from the [Apache mirrors][mirror].
-* Source archive: [knox-0.4.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha], [MD5 digest][src-md5])
-* Binary archive: [knox-0.4.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha], [MD5 digest][bin-md5])
+* Source archive: [knox-0.5.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha], [MD5 digest][src-md5])
+* Binary archive: [knox-0.5.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha], [MD5 digest][bin-md5])
-[src-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip
-[src-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.sha
-[src-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.asc
-[src-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.md5
-[bin-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip
-[bin-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.asc
-[bin-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.sha
-[bin-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.md5
+[src-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.4.0-src.zip
+[src-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.sha
+[src-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.asc
+[src-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0-src.zip.md5
+[bin-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip
+[bin-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.asc
+[bin-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.sha
+[bin-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.5.0/knox-0.5.0.zip.md5
Apache Knox Gateway releases are available under the [Apache License, Version 2.0][asl].
See the NOTICE file contained in each release artifact for applicable copyright attribution notices.
@@ -80,19 +80,19 @@ Make sure you get these files from the m
Then verify the signatures using one of the methods below.
% pgpk -a KEYS
- % pgpv knox-0.4.0.zip.asc
+ % pgpv knox-0.5.0.zip.asc
or
% pgp -ka KEYS
- % pgp knox-0.4.0.zip.asc
+ % pgp knox-0.5.0.zip.asc
or
% gpg --import KEYS
- % gpg --verify knox-0.4.0.zip.asc
+ % gpg --verify knox-0.5.0.zip.asc
-### 4 - Start Hadoop virtual machine ###
+### 5 - Start Hadoop virtual machine ###
Start the Hadoop virtual machine.