You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2013/11/14 23:03:41 UTC
svn commit: r1542103 - in /incubator/knox: site/
site/books/knox-incubating-0-3-0/ trunk/books/0.3.0/
Author: kminder
Date: Thu Nov 14 22:03:41 2013
New Revision: 1542103
URL: http://svn.apache.org/r1542103
Log:
Add Limitations section and some troubleshooting stubs.
Added:
incubator/knox/trunk/books/0.3.0/book_limitations.md
- copied, changed from r1542068, incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
Modified:
incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
incubator/knox/site/index.html
incubator/knox/site/issue-tracking.html
incubator/knox/site/license.html
incubator/knox/site/mail-lists.html
incubator/knox/site/project-info.html
incubator/knox/site/team-list.html
incubator/knox/trunk/books/0.3.0/book.md
incubator/knox/trunk/books/0.3.0/book_getting-started.md
incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
incubator/knox/trunk/books/0.3.0/quick_start.md
Modified: incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html (original)
+++ incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html Thu Nov 14 22:03:41 2013
@@ -16,7 +16,7 @@
--><p><link href="book.css" rel="stylesheet"/></p>
<div id="logo" style="width:100%; text-align:center">
<!--img src="knox-logo.gif" alt="Knox"/-->
-</div><p><br> <img src="knox-logo.gif" alt="Knox"/> <img src="apache-incubator-logo.png" align="right" alt="Incubator"/></p><h1><a id="Apache+Knox+Gateway+0.3.0+(Incubator)+User's+Guide"></a>Apache Knox Gateway 0.3.0 (Incubator) User’s Guide</h1><h2><a id="Table+Of+Contents"></a>Table Of Contents</h2>
+</div><p><br> <img src="knox-logo.gif" alt="Knox"/> <img src="apache-incubator-logo.png" align="right" alt="Incubator"/></p><h1><a id="Apache+Knox+Gateway+0.3.x+(Incubator)+User's+Guide"></a>Apache Knox Gateway 0.3.x (Incubator) User’s Guide</h1><h2><a id="Table+Of+Contents"></a>Table Of Contents</h2>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#Quick+Start">Quick Start</a></li>
@@ -44,6 +44,7 @@
<li><a href="#HBase">HBase</a></li>
<li><a href="#Hive">Hive</a></li>
</ul></li>
+ <li><a href="#Limitations">Limitations</a></li>
<li><a href="#Troubleshooting">Troubleshooting</a></li>
<li><a href="#Export+Controls">Export Controls</a></li>
</ul><h2><a id="Introduction"></a>Introduction</h2><p>The Apache Knox Gateway is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The gateway runs as a server (or cluster of servers) that provide centralized access to one or more Hadoop clusters. In general the goals of the gateway are as follows:</p>
@@ -92,17 +93,23 @@
</code></pre><p>or</p>
<pre><code>sudo rpm -ihv knox-incubating-{VERSION}.rpm
</code></pre><h3><a id="6+-+Start+LDAP+embedded+in+Knox"></a>6 - Start LDAP embedded in Knox</h3><p>Knox comes with an LDAP server for demonstration purposes.</p>
-<pre><code>java -jar {GATEWAY_HOME}/bin/ldap.jar conf &
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/ldap.jar conf &
</code></pre><h3><a id="7+-+Start+Knox"></a>7 - Start Knox</h3><p>The gateway can be started in one of two ways, as java -jar or with a shell script.</p><h6><a id="Starting+via+Java"></a>Starting via Java</h6><p>This is the simplest way to start the gateway. Starting this way will result in all logging being written directly to standard output.</p>
-<pre><code>java -jar {GATEWAY_HOME}/bin/gateway.jar
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/gateway.jar
</code></pre><p>Upon start, Knox server will prompt you for the master secret (i.e. password). This secret is used to secure artifacts used by the gateway server for things like SSL and credential/password aliasing. This secret will have to be entered at startup unless you choose to persist it.</p><h6><a id="Starting+via+script+(*nix+only)"></a>Starting via script (*nix only)</h6><p>Run the setup command with root privileges.</p>
-<pre><code>sudo {GATEWAY_HOME}/bin/gateway.sh setup
+<pre><code>cd {GATEWAY_HOME}
+sudo bin/gateway.sh setup
</code></pre><p>The server will prompt you for the master secret (i.e. password).</p><p>The server can then be started without root privileges using this command.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh start
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh start
</code></pre><p>When starting the gateway this way the process will be run in the backgroud. The log output is written into the directory /var/log/knox. In addition a PID (process ID) is written into /var/run/knox.</p><p>In order to stop a gateway that was started with the script use this command.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh stop
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh stop
</code></pre><p>If for some reason the gateway is stopped other than by using the command above you may need to clear the tracking PID.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh clean
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh clean
</code></pre><p><strong>NOTE: This command will also clear any log output in /var/log/knox so use this with caution.</strong></p><h3><a id="8+-+Do+Hadoop+with+Knox"></a>8 - Do Hadoop with Knox</h3><h4><a id="Put+a+file+in+HDFS+via+Knox."></a>Put a file in HDFS via Knox.</h4><h4><a id="CAT+a+file+in+HDFS+via+Knox."></a>CAT a file in HDFS via Knox.</h4><h4><a id="Invoke+the+LISTSATUS+operation+on+WebHDFS+via+the+gateway."></a>Invoke the LISTSATUS operation on WebHDFS via the gateway.</h4><p>This will return a directory listing of the root (i.e. /) directory of HDFS.</p>
<pre><code>curl -i -k -u guest:guest-password -X GET \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
@@ -270,7 +277,7 @@ Server: Jetty(6.1.26)
<td><img src="error.png" alt="n"/> </td>
</tr>
</tbody>
-</table><h3><a id="Sandbox+Configuration"></a>Sandbox Configuration</h3><p>TODO</p><h3><a id="More+Examples"></a>More Examples</h3><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
+</table><h3><a id="More+Examples"></a>More Examples</h3><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
<ul>
<li><a href="#WebHDFS+Examples">WebHDFS Examples</a></li>
<li><a href="#WebHCat+Examples">WebHCat Examples</a></li>
@@ -2255,15 +2262,28 @@ connection.close();
2012-02-03 --- 18:35:34 --- SampleClass6 --- [TRACE]
2012-02-03 --- 18:35:34 --- SampleClass2 --- [DEBUG]
...
-</code></pre><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Connection+Errors"></a>Connection Errors</h3><p>TODO - Explain how to debug connection errors.</p><h3><a id="Enabling+Logging"></a>Enabling Logging</h3><p>The <code>log4j.properties</code> files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of the logging done by Knox. The Knox server must be restarted in order for these changes to take effect. There are various useful loggers pre-populated but commented out.</p>
+</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT Request Payload Size Restriction</h3><p>With one exception there are no know size limits for requests or responses payloads that pass through the gateway. The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego. There is an undocumented configuration setting to modify this limit’s value if required. In the future this will be made more easily configuration and at that time it will be documented.</p><h3><a id="LDAP+Groups+Acquisition"></a>LDAP Groups Acquisition</h3><p>The LDAP authenticator currently does not “out of the box” support the acquisition of
group information. This can be addressed by implementing a custom Shiro Realm extension. Building this into the default implementation is on the roadmap.</p><h3><a id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups that are acquired via Identity Assertion Group Principal Mapping are not propigated to the Hadoop services. Therefore groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding Logs</h3><p>When things aren’t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output is written directly to the console. If you want to ca
pture that output you will need to redirect the console output to a file using OS specific techniques.</p>
+<pre><code>java -jar bin/gateway.jar > gateway.log
+</code></pre><h4><a id="bin/gateway.sh+start"></a>bin/gateway.sh start</h4><p>When the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out and /var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a id="Increasing+Logging"></a>Increasing Logging</h3><p>The <code>log4j.properties</code> files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of the logging done by Knox. The Knox server must be restarted in order for these changes to take effect. There are various useful loggers pre-populated but commented out.</p>
<pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging of Apache Knox itself.
log4j.logger.org.apache.shiro=DEBUG # Use this logger to increase the debugging of Apache Shiro.
log4j.logger.org.apache.http=DEBUG # Use this logger to increase the debugging of Apache HTTP components.
log4j.logger.org.apache.http.client=DEBUG # Use this logger to increase the debugging of Apache HTTP client component.
log4j.logger.org.apache.http.headers=DEBUG # Use this logger to increase the debugging of Apache HTTP header.
log4j.logger.org.apache.http.wire=DEBUG # Use this logger to increase the debugging of Apache HTTP wire traffic.
-</code></pre><h3><a id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs can be filed using <a href="https://issues.apache.org/jira/browse/KNOX">Jira</a>. Please include the results of this command below in the Environment section. Also include the version of Hadoop being used in the same section.</p>
-<pre><code>java -jar bin/gateway.jar -version
+</code></pre><h3><a id="LDAP+Server+Connectivity+Issues"></a>LDAP Server Connectivity Issues</h3><p>If the gateway cannot contact the configured LDAP server you will see errors in the gateway diagnostic output.</p>
+<pre><code>TODO:Kevin - What does it look like when the LDAP server isn't running.
+</code></pre><p>Resolving this will require ensuring that the LDAP server is running and that connection information is correct. The LDAP server connection information is configured in the cluster’s topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).</p><h3><a id="Hadoop+Cluster+Connectivity+Issues"></a>Hadoop Cluster Connectivity Issues</h3><p>If the gateway cannot contact one of the services in the configured Hadoop cluster you will see errors in the gateway diagnostic output.</p>
+<pre><code>TODO:Kevin - What does it look like when the Sandbox isn't running.
+</code></pre><p>Resolving this will require ensuring that the Hadoop services are running and that connection information is correct. Basic Hadoop connectivity can be evaluated using cURL as described elsewhere. Otherwise the Hadoop cluster connection information is configured in the cluster’s topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).</p><h3><a id="Check+Hadoop+Cluster+Access+via+cURL"></a>Check Hadoop Cluster Access via cURL</h3><p>When you are experiencing connectivity issue it can be helpful to “bypass” the gateway and invoke the Hadoop REST APIs directly. This can easily be done using the cURL command line utility or many other REST/HTTP clients. Exactly how to use cURL depends on the configuration of your Hadoop cluster. In general however you will use a command line the one that follows.</p>
+<pre><code>curl -ikv -X GET 'http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS'
+</code></pre><p>If you are using Sandbox the WebHDFS or NameNode port will be mapped to localhost so this command can be used.</p>
+<pre><code>curl -ikv -X GET 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+</code></pre><p>If you are using a cluster secured with Kerberos you will need to have used <code>kinit</code> to authenticate to the KDC. Then the command below should verify that WebHDFS in the Hadoop cluster is accessible.</p>
+<pre><code>curl -ikv --negotiate -u : -X 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+</code></pre><h3><a id="Authentication+Issues"></a>Authentication Issues</h3><p>TODO:Kevin - What does it look like when the username/password don’t match what is in LDAP?</p><h3><a id="Hostname+Resolution+Issues"></a>Hostname Resolution Issues</h3><p>TODO:Kevin - What does it look like when host mapping is enabled and shouldn’t be or vice versa.</p><h3><a id="Job+Submission+Issues+-+HDFS+Home+Directories"></a>Job Submission Issues - HDFS Home Directories</h3><p>TODO:Dilli - What does it look like if the LDAP authenticated user doesn’t have a HDFS home directory and submits a job.</p><h3><a id="Job+Submission+Issues+-+OS+Accounts"></a>Job Submission Issues - OS Accounts</h3><p>TODO:Dilli - What does it look like if the LDAP authenticated user submits a job but doesn’t have an OS account.</p><h3><a id="HBase+Issues"></a>HBase Issues</h3><p>TODO:Kevin - What does it look like when HBase/Stargate hangs and how do you fix it.</p><h3><a id="SSL+Certificate+Issues"
></a>SSL Certificate Issues</h3><p>TODO:Larry - What does it look like when a client doesn’t trust the gateway’s SSL identity certificate?</p><h3><a id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs can be filed using <a href="https://issues.apache.org/jira/browse/KNOX">Jira</a>. Please include the results of this command below in the Environment section. Also include the version of Hadoop being used in the same section.</p>
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/gateway.jar -version
</code></pre><h2><a id="Export+Controls"></a>Export Controls</h2><p>Apache Knox Gateway includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country’s laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <a href="http://www.wassenaar.org">http://www.wassenaar.org</a> for more information.</p><p>The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC
Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.</p><p>The following provides more details on the included cryptographic software:</p>
<ul>
<li>Apache Knox Gateway uses the ApacheDS which in turn uses Bouncy Castle generic encryption libraries.</li>
Modified: incubator/knox/site/index.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/index.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/index.html (original)
+++ incubator/knox/site/index.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/issue-tracking.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/issue-tracking.html (original)
+++ incubator/knox/site/issue-tracking.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/license.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/license.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/license.html (original)
+++ incubator/knox/site/license.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/mail-lists.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/mail-lists.html (original)
+++ incubator/knox/site/mail-lists.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/project-info.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/project-info.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/project-info.html (original)
+++ incubator/knox/site/project-info.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/site/team-list.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/team-list.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/team-list.html (original)
+++ incubator/knox/site/team-list.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20131104" />
+ <meta name="Date-Revision-yyyymmdd" content="20131114" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2013-11-04</span>
+ | <span id="publishDate">Last Published: 2013-11-14</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: incubator/knox/trunk/books/0.3.0/book.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book.md (original)
+++ incubator/knox/trunk/books/0.3.0/book.md Thu Nov 14 22:03:41 2013
@@ -24,7 +24,7 @@
<img src="knox-logo.gif" alt="Knox"/>
<img src="apache-incubator-logo.png" align="right" alt="Incubator"/>
-# Apache Knox Gateway 0.3.0 (Incubator) User's Guide #
+# Apache Knox Gateway 0.3.x (Incubator) User's Guide #
## Table Of Contents ##
@@ -48,6 +48,7 @@
* #[Oozie]
* #[HBase]
* #[Hive]
+* #[Limitations]
* #[Troubleshooting]
* #[Export Controls]
@@ -72,6 +73,7 @@ In general the goals of the gateway are
<<book_gateway-details.md>>
<<book_client-details.md>>
<<book_service-details.md>>
+<<book_limitations.md>>
<<book_troubleshooting.md>>
Modified: incubator/knox/trunk/books/0.3.0/book_getting-started.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_getting-started.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_getting-started.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_getting-started.md Thu Nov 14 22:03:41 2013
@@ -93,10 +93,6 @@ Only more recent versions of some Hadoop
| | 0.12.0 | ![n] | ![n] |
-### Sandbox Configuration ###
-
-TODO
-
### More Examples ###
These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.
Copied: incubator/knox/trunk/books/0.3.0/book_limitations.md (from r1542068, incubator/knox/trunk/books/0.3.0/book_troubleshooting.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_limitations.md?p2=incubator/knox/trunk/books/0.3.0/book_limitations.md&p1=incubator/knox/trunk/books/0.3.0/book_troubleshooting.md&r1=1542068&r2=1542103&rev=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_troubleshooting.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_limitations.md Thu Nov 14 22:03:41 2013
@@ -15,32 +15,28 @@
limitations under the License.
--->
-## Troubleshooting ##
+## Limitations ##
-### Connection Errors ###
-TODO - Explain how to debug connection errors.
+### Secure Oozie POST/PUT Request Payload Size Restriction ###
+With one exception there are no know size limits for requests or responses payloads that pass through the gateway.
+The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster.
+In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster.
+This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego.
+There is an undocumented configuration setting to modify this limit's value if required.
+In the future this will be made more easily configuration and at that time it will be documented.
-### Enabling Logging ###
-The `log4j.properties` files `{GATEWAY_HOME}/conf` can be used to change the granularity of the logging done by Knox.
-The Knox server must be restarted in order for these changes to take effect.
-There are various useful loggers pre-populated but commented out.
+### LDAP Groups Acquisition ###
- log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging of Apache Knox itself.
- log4j.logger.org.apache.shiro=DEBUG # Use this logger to increase the debugging of Apache Shiro.
- log4j.logger.org.apache.http=DEBUG # Use this logger to increase the debugging of Apache HTTP components.
- log4j.logger.org.apache.http.client=DEBUG # Use this logger to increase the debugging of Apache HTTP client component.
- log4j.logger.org.apache.http.headers=DEBUG # Use this logger to increase the debugging of Apache HTTP header.
- log4j.logger.org.apache.http.wire=DEBUG # Use this logger to increase the debugging of Apache HTTP wire traffic.
+The LDAP authenticator currently does not "out of the box" support the acquisition of group information.
+This can be addressed by implementing a custom Shiro Realm extension.
+Building this into the default implementation is on the roadmap.
-### Filing Bugs ###
+### Group Membership Propagation ###
-Bugs can be filed using [Jira][jira].
-Please include the results of this command below in the Environment section.
-Also include the version of Hadoop being used in the same section.
-
- java -jar bin/gateway.jar -version
+Groups that are acquired via Identity Assertion Group Principal Mapping are not propigated to the Hadoop services.
+Therefore groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.
Modified: incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_troubleshooting.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_troubleshooting.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_troubleshooting.md Thu Nov 14 22:03:41 2013
@@ -17,12 +17,25 @@
## Troubleshooting ##
-### Connection Errors ###
+### Finding Logs ###
-TODO - Explain how to debug connection errors.
+When things aren't working the first thing you need to do is examine the diagnostic logs.
+Depending upon how you are running the gateway these diagnostic logs will be output to different locations.
+#### java -jar bin/gateway.jar ####
-### Enabling Logging ###
+When the gateway is run this way the diagnostic output is written directly to the console.
+If you want to capture that output you will need to redirect the console output to a file using OS specific techniques.
+
+ java -jar bin/gateway.jar > gateway.log
+
+#### bin/gateway.sh start ####
+
+When the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out and /var/log/knox/knox.err.
+Typically only knox.out will have content.
+
+
+### Increasing Logging ###
The `log4j.properties` files `{GATEWAY_HOME}/conf` can be used to change the granularity of the logging done by Knox.
The Knox server must be restarted in order for these changes to take effect.
@@ -36,11 +49,82 @@ There are various useful loggers pre-pop
log4j.logger.org.apache.http.wire=DEBUG # Use this logger to increase the debugging of Apache HTTP wire traffic.
+### LDAP Server Connectivity Issues ###
+
+If the gateway cannot contact the configured LDAP server you will see errors in the gateway diagnostic output.
+
+ TODO:Kevin - What does it look like when the LDAP server isn't running.
+
+Resolving this will require ensuring that the LDAP server is running and that connection information is correct.
+The LDAP server connection information is configured in the cluster's topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).
+
+
+### Hadoop Cluster Connectivity Issues ###
+
+If the gateway cannot contact one of the services in the configured Hadoop cluster you will see errors in the gateway diagnostic output.
+
+ TODO:Kevin - What does it look like when the Sandbox isn't running.
+
+Resolving this will require ensuring that the Hadoop services are running and that connection information is correct.
+Basic Hadoop connectivity can be evaluated using cURL as described elsewhere.
+Otherwise the Hadoop cluster connection information is configured in the cluster's topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).
+
+
+### Check Hadoop Cluster Access via cURL ###
+
+When you are experiencing connectivity issue it can be helpful to "bypass" the gateway and invoke the Hadoop REST APIs directly.
+This can easily be done using the cURL command line utility or many other REST/HTTP clients.
+Exactly how to use cURL depends on the configuration of your Hadoop cluster.
+In general however you will use a command line the one that follows.
+
+ curl -ikv -X GET 'http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS'
+
+If you are using Sandbox the WebHDFS or NameNode port will be mapped to localhost so this command can be used.
+
+ curl -ikv -X GET 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+
+If you are using a cluster secured with Kerberos you will need to have used `kinit` to authenticate to the KDC.
+Then the command below should verify that WebHDFS in the Hadoop cluster is accessible.
+
+ curl -ikv --negotiate -u : -X 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+
+
+### Authentication Issues ###
+
+TODO:Kevin - What does it look like when the username/password don't match what is in LDAP?
+
+
+### Hostname Resolution Issues ###
+
+TODO:Kevin - What does it look like when host mapping is enabled and shouldn't be or vice versa.
+
+
+### Job Submission Issues - HDFS Home Directories ###
+
+TODO:Dilli - What does it look like if the LDAP authenticated user doesn't have a HDFS home directory and submits a job.
+
+
+### Job Submission Issues - OS Accounts ###
+
+TODO:Dilli - What does it look like if the LDAP authenticated user submits a job but doesn't have an OS account.
+
+
+### HBase Issues ###
+
+TODO:Kevin - What does it look like when HBase/Stargate hangs and how do you fix it.
+
+
+### SSL Certificate Issues ###
+
+TODO:Larry - What does it look like when a client doesn't trust the gateway's SSL identity certificate?
+
+
### Filing Bugs ###
Bugs can be filed using [Jira][jira].
Please include the results of this command below in the Environment section.
Also include the version of Hadoop being used in the same section.
+ cd {GATEWAY_HOME}
java -jar bin/gateway.jar -version
Modified: incubator/knox/trunk/books/0.3.0/quick_start.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/quick_start.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/quick_start.md (original)
+++ incubator/knox/trunk/books/0.3.0/quick_start.md Thu Nov 14 22:03:41 2013
@@ -139,7 +139,8 @@ or
Knox comes with an LDAP server for demonstration purposes.
- java -jar {GATEWAY_HOME}/bin/ldap.jar conf &
+ cd {GATEWAY_HOME}
+ java -jar bin/ldap.jar conf &
### 7 - Start Knox ###
@@ -152,7 +153,8 @@ The gateway can be started in one of two
This is the simplest way to start the gateway.
Starting this way will result in all logging being written directly to standard output.
- java -jar {GATEWAY_HOME}/bin/gateway.jar
+ cd {GATEWAY_HOME}
+ java -jar bin/gateway.jar
Upon start, Knox server will prompt you for the master secret (i.e. password).
@@ -164,13 +166,15 @@ This secret will have to be entered at s
Run the setup command with root privileges.
- sudo {GATEWAY_HOME}/bin/gateway.sh setup
+ cd {GATEWAY_HOME}
+ sudo bin/gateway.sh setup
The server will prompt you for the master secret (i.e. password).
The server can then be started without root privileges using this command.
- {GATEWAY_HOME}/bin/gateway.sh start
+ cd {GATEWAY_HOME}
+ bin/gateway.sh start
When starting the gateway this way the process will be run in the backgroud.
The log output is written into the directory /var/log/knox.
@@ -178,11 +182,13 @@ In addition a PID (process ID) is writte
In order to stop a gateway that was started with the script use this command.
- {GATEWAY_HOME}/bin/gateway.sh stop
+ cd {GATEWAY_HOME}
+ bin/gateway.sh stop
If for some reason the gateway is stopped other than by using the command above you may need to clear the tracking PID.
- {GATEWAY_HOME}/bin/gateway.sh clean
+ cd {GATEWAY_HOME}
+ bin/gateway.sh clean
__NOTE: This command will also clear any log output in /var/log/knox so use this with caution.__