You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2015/07/22 20:20:28 UTC
svn commit: r1692312 - in /knox: site/books/knox-0-7-0/user-guide.html
trunk/books/0.7.0/book_service-details.md trunk/books/0.7.0/knox_cli.md
trunk/books/0.7.0/service_service_test.md
Author: kminder
Date: Wed Jul 22 18:20:28 2015
New Revision: 1692312
URL: http://svn.apache.org/r1692312
Log:
KNOX-549: Test service connections through Knox with Knox CLI
Added:
knox/trunk/books/0.7.0/service_service_test.md
Modified:
knox/site/books/knox-0-7-0/user-guide.html
knox/trunk/books/0.7.0/book_service-details.md
knox/trunk/books/0.7.0/knox_cli.md
Modified: knox/site/books/knox-0-7-0/user-guide.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-7-0/user-guide.html?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/site/books/knox-0-7-0/user-guide.html (original)
+++ knox/site/books/knox-0-7-0/user-guide.html Wed Jul 22 18:20:28 2015
@@ -707,6 +707,40 @@ ip-10-39-107-209.ec2.internal
<td>Optional; Print extra debug info on failed authentication</td>
</tr>
</tbody>
+</table><h4><a id="Gateway+Service+Test"></a>Gateway Service Test</h4><h5><a id="`bin/knoxcli.sh+service-test+[--cluster+c]+[--hostname+hostname]+[--port+port]+[--u+username]+[--p+password]+[--d]+[--help]`"></a><code>bin/knoxcli.sh service-test [--cluster c] [--hostname hostname] [--port port] [--u username] [--p password] [--d] [--help]</code></h5><p>This will test a topology configuration’s ability to connect to multiple hadoop services. Each service found in a topology will be tested with multiple URLs. Results are printed to the console in JSON format..</p>
+<table>
+ <thead>
+ <tr>
+ <th>argument </th>
+ <th>description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>--cluster </td>
+ <td>Required; name of cluster for which you want to test authentication</td>
+ </tr>
+ <tr>
+ <td>--hostname </td>
+ <td>Required; hostname of the cluster currently running on the machine</td>
+ </tr>
+ <tr>
+ <td>--port </td>
+ <td>Optional; port that the cluster is running on. If not supplied CLI will try to read config files to find the port.</td>
+ </tr>
+ <tr>
+ <td>--u </td>
+ <td>Required; username to authorize against Hadoop services</td>
+ </tr>
+ <tr>
+ <td>--p </td>
+ <td>Required; password to match username</td>
+ </tr>
+ <tr>
+ <td>--d </td>
+ <td>Optional; Print extra debug info on failed authentication</td>
+ </tr>
+ </tbody>
</table><h3><a id="Admin+API"></a>Admin API</h3><p>Access to the administrator functions of Knox are provided by the Admin REST API.</p><h4><a id="Admin+API+URL"></a>Admin API URL</h4><p>The URL mapping for the Knox Admin API is simple:</p>
<table>
<tbody>
@@ -3526,7 +3560,201 @@ curl -ik -b ~/cookiejar.txt -c ~/cookiej
curl -ik -b ~/cookiejar.txt -c ~/cookiejar.txt -u guest:guest-password -H 'x-csrf-token:{token-value}' -X POST \
http://localhost:8744/api/v1/topology/{id}/kill/0
-</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT Request Payload Size Restriction</h3><p>With one exception there are no known size limits for requests or responses payloads that pass through the gateway. The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego. There is an undocumented configuration setting to modify this limit’s value if required. In the future this will be made more easily configuration and at that time it will be documented.</p><h3><a id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups that are acquired via Shiro Group Lookup and/or Identity Assertion Group Princip
al Mapping are not propagated to the Hadoop services. Therefore, groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding Logs</h3><p>When things aren’t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output is written directly to the console. If you want to capture that output you will need to redirect the console output to a file using OS specific techniques.</p>
+</code></pre><h3><a id="Service+Test+API"></a>Service Test API</h3><p>The gateway supports a Service Test API that can be used to test Knox’s ability to connect to each of the different Hadoop services via a simeple HTTP GET request. To be able to access this API one must add the following line into the topology for which you wish to run the service test.</p>
+<pre><code><service>
+ <role>SERVICE-TEST</role>
+</service>
+</code></pre><p>After adding the above to a topology, you can make a cURL request with the following structure</p>
+<pre><code>curl -i -k "https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password"
+</code></pre><p>An alternate method of providing credentials:</p>
+<pre><code>curl -i -k -u guest:guest-password https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
+</code></pre><p>Below is an example response. The gateway is also capable of returning XML if specified in the request’s “Accept” HTTP header.</p>
+<pre><code>{
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+
+ ]
+ }
+ }
+}
+</code></pre><p>We can see that this service-test makes HTTP requests to each of the services through Knox using the specified topology. The test will only make calls to those services that have entries within the topology file.</p><h5><a id="Adding+and+Changing+test+URLs"></a>Adding and Changing test URLs</h5><p>URLs for each service are stored in <code>{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml</code>. Each <code><testURL></code> element represents a service resource that will be tested if the service is set up in the topology. You can add or remove these from teh service.xml files. Just note if you add URL’s there is no guarantee in the order they will be tested. All default URLs have been tested and work on various clusters. If a new URL is added and doesn’t respond in a way the user expects then it is up to the user to determine whether the URL is correct or not.</p><h5><a id="Some+important+things+to+note:"></a>Some important th
ings to note:</h5>
+<ul>
+ <li>In the first cURL request, the quotes are necessary around the URL or else a command line terminal will not include the <code>&password</code> query parameter in the request.</li>
+ <li>This API call does not require any credentials to receive a response from Knox, but expect to receive 401 responses from each of the services if none are provided.</li>
+</ul><h2><a id="Limitations"></a>Limitations</h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT Request Payload Size Restriction</h3><p>With one exception there are no known size limits for requests or responses payloads that pass through the gateway. The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego. There is an undocumented configuration setting to modify this limit’s value if required. In the future this will be made more easily configuration and at that time it will be documented.</p><h3><a id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups that are acquired via Shiro Group Lookup and/or Identity Assertion Group Principal Mappi
ng are not propagated to the Hadoop services. Therefore, groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding Logs</h3><p>When things aren’t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output is written directly to the console. If you want to capture that output you will need to redirect the console output to a file using OS specific techniques.</p>
<pre><code>java -jar bin/gateway.jar > gateway.log
</code></pre><h4><a id="bin/gateway.sh+start"></a>bin/gateway.sh start</h4><p>When the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out and /var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a id="Increasing+Logging"></a>Increasing Logging</h3><p>The <code>log4j.properties</code> files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of the logging done by Knox. The Knox server must be restarted in order for these changes to take effect. There are various useful loggers pre-populated but commented out.</p>
<pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging of Apache Knox itself.
Modified: knox/trunk/books/0.7.0/book_service-details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book_service-details.md?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book_service-details.md (original)
+++ knox/trunk/books/0.7.0/book_service-details.md Wed Jul 22 18:20:28 2015
@@ -84,3 +84,213 @@ Therefore each request via cURL will res
<<service_storm.md>>
+### Service Test API
+
+The gateway supports a Service Test API that can be used to test Knox's ability to connect to each of the different Hadoop services via a simeple HTTP GET request. To be able to access this API one must add the following line into the topology for which you wish to run the service test.
+
+ <service>
+ <role>SERVICE-TEST</role>
+ </service>
+
+After adding the above to a topology, you can make a cURL request with the following structure
+
+ curl -i -k "https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password"
+
+An alternate method of providing credentials:
+
+ curl -i -k -u guest:guest-password https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
+
+Below is an example response. The gateway is also capable of returning XML if specified in the request's "Accept" HTTP header.
+
+ {
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+
+ ]
+ }
+ }
+ }
+
+
+We can see that this service-test makes HTTP requests to each of the services through Knox using the specified topology. The test will only make calls to those services that have entries within the topology file.
+
+##### Adding and Changing test URLs
+
+URLs for each service are stored in `{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`. Each `<testURL>` element represents a service resource that will be tested if the service is set up in the topology. You can add or remove these from teh service.xml files. Just note if you add URL's there is no guarantee in the order they will be tested. All default URLs have been tested and work on various clusters. If a new URL is added and doesn't respond in a way the user expects then it is up to the user to determine whether the URL is correct or not.
+
+##### Some important things to note:
+ - In the first cURL request, the quotes are necessary around the URL or else a command line terminal will not include the `&password` query parameter in the request.
+ - This API call does not require any credentials to receive a response from Knox, but expect to receive 401 responses from each of the services if none are provided.
Modified: knox/trunk/books/0.7.0/knox_cli.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/knox_cli.md?rev=1692312&r1=1692311&r2=1692312&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/knox_cli.md (original)
+++ knox/trunk/books/0.7.0/knox_cli.md Wed Jul 22 18:20:28 2015
@@ -110,4 +110,19 @@ This command will test a given topology'
argument | description
---------|-----------
\-\-cluster | Required; name of cluster for which you want to test authentication
+\-\-d | Optional; Print extra debug info on failed authentication
+
+
+#### Gateway Service Test ####
+##### `bin/knoxcli.sh service-test [--cluster c] [--hostname hostname] [--port port] [--u username] [--p password] [--d] [--help]` ####
+
+This will test a topology configuration's ability to connect to multiple hadoop services. Each service found in a topology will be tested with multiple URLs. Results are printed to the console in JSON format..
+
+argument | description
+---------|-----------
+\-\-cluster | Required; name of cluster for which you want to test authentication
+\-\-hostname | Required; hostname of the cluster currently running on the machine
+\-\-port | Optional; port that the cluster is running on. If not supplied CLI will try to read config files to find the port.
+\-\-u | Required; username to authorize against Hadoop services
+\-\-p | Required; password to match username
\-\-d | Optional; Print extra debug info on failed authentication
\ No newline at end of file
Added: knox/trunk/books/0.7.0/service_service_test.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/service_service_test.md?rev=1692312&view=auto
==============================================================================
--- knox/trunk/books/0.7.0/service_service_test.md (added)
+++ knox/trunk/books/0.7.0/service_service_test.md Wed Jul 22 18:20:28 2015
@@ -0,0 +1,221 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--->
+
+### Service Test ###
+
+The gateway supports a Service Test API that can be used to test Knox's ability to connect to each of the different Hadoop services via a simple HTTP GET request. To be able to access this API one must add the following entry into the topology for which you wish to run the service test.
+
+ <service>
+ <role>SERVICE-TEST</role>
+ </service>
+
+After adding the above to a topology, you can make a cURL request with the following structure
+
+ curl -i -k -u guest:guest-password https://{gateway-host}:{gateway-port}/{gateway-path}/{topology-name}/service-test
+
+Below is an example response. The gateway is also capable of returning XML if specified in the request's "Accept" HTTP header.
+
+ {
+ "serviceTestWrapper": {
+ "Tests": {
+ "ServiceTest": [
+ {
+ "serviceName": "WEBHDFS",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHCAT",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "OOZIE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "WEBHBASE",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "RESOURCEMANAGER",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "FALCON",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ },
+ {
+ "serviceName": "STORM",
+ "requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary",
+ "responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
+ "httpCode": 200,
+ "message": "Request sucessful."
+ }
+ ]
+ },
+ "messages": {
+ "message": [
+ ]
+ }
+ }
+ }
+
+
+We can see that this service-test makes HTTP requests to each of the services through Knox using the specified topology. The test will only make calls to those services that have entries within the topology file.
+
+##### Adding and Changing test URLs
+
+URLs for each service are stored in `{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`. Each `<testURL>` element represents a service resource that will be tested if the service is set up in the topology. You can add or remove these from teh service.xml files. Just note if you add URL's there is no guarantee in the order they will be tested. All default URLs have been tested and work on various clusters. If a new URL is added and doesn't respond in a way the user expects then it is up to the user to determine whether the URL is correct or not.
+
+##### Some important things to note:
+ - This API call does not require any credentials to receive a response from Knox, but expect to receive 401 responses from each of the services if none are provided.