You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2013/09/25 23:40:30 UTC
svn commit: r1526308 [1/2] - in /incubator/knox:
site/books/knox-incubating-0-3-0/ trunk/books/0.3.0/ trunk/books/static/
Author: kminder
Date: Wed Sep 25 21:40:30 2013
New Revision: 1526308
URL: http://svn.apache.org/r1526308
Log:
Added HBase and Hive.
Added:
incubator/knox/trunk/books/0.3.0/sandbox.md
Modified:
incubator/knox/site/books/knox-incubating-0-3-0/book.css
incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
incubator/knox/trunk/books/0.3.0/book.md
incubator/knox/trunk/books/0.3.0/client.md
incubator/knox/trunk/books/0.3.0/config.md
incubator/knox/trunk/books/0.3.0/hbase.md
incubator/knox/trunk/books/0.3.0/hive.md
incubator/knox/trunk/books/static/book.css
Modified: incubator/knox/site/books/knox-incubating-0-3-0/book.css
URL: http://svn.apache.org/viewvc/incubator/knox/site/books/knox-incubating-0-3-0/book.css?rev=1526308&r1=1526307&r2=1526308&view=diff
==============================================================================
--- incubator/knox/site/books/knox-incubating-0-3-0/book.css (original)
+++ incubator/knox/site/books/knox-incubating-0-3-0/book.css Wed Sep 25 21:40:30 2013
@@ -95,7 +95,10 @@ h6 {
color: #777777;
font-size: 14px; }
-p, blockquote, ul, ol, dl, li, table, pre {
+ul {
+ margin: 0px 0; }
+
+p, blockquote, ol, dl, li, table, pre {
margin: 15px 0; }
hr {
Modified: incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html?rev=1526308&r1=1526307&r2=1526308&view=diff
==============================================================================
--- incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html (original)
+++ incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html Wed Sep 25 21:40:30 2013
@@ -20,6 +20,7 @@
<li><a href="#Installation">Installation</a></li>
<li><a href="#Getting+Started">Getting Started</a></li>
<li><a href="#Supported+Services">Supported Services</a></li>
+ <li><a href="#Sandbox+Configuration">Sandbox Configuration</a></li>
<li><a href="#Usage+Examples">Usage Examples</a></li>
<li><a href="#Gateway+Details">Gateway Details</a>
<ul>
@@ -40,7 +41,9 @@
<li><a href="#Trouble+Shooting">Trouble Shooting</a></li>
<li><a href="#Export+Controls">Export Controls</a></li>
<li><a href="#Release+Verification">Release Verification</a></li>
-</ul><h2><a id="Introduction"></a>Introduction</h2><p>TODO</p><h2><a id="Requirements"></a>Requirements</h2><h3>Java</h3><p>Java 1.6 or later is required for the Knox Gateway runtime. Use the command below to check the version of Java installed on the system where Knox will be running.</p><p>{code} java -version {code}</p><h3>Hadoop</h3><p>An an existing Hadoop 1.x or 2.x cluster is required for Knox to protect. One of the easiest ways to ensure this it to utilize a HDP Sandbox VM. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration. Currently if this Hadoop cluster is secured with Kerberos only WebHDFS will work and additional configuration is required.</p><p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow
assume that the Gateway is <em>not</em> collocated with the Hadoop clusters themselves and (most importantly) that the hostnames and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running. All of the instructions and samples are tailored to work “out of the box” against a Hortonworks Sandbox 2.x VM.</p><p>This release of the Apache Knox Gateway has been tested against the <a href="http://hortonworks.com/products/hortonworks-sandbox/">Hortonworks Sandbox 2.0</a>.</p><h2><a id="Download"></a>Download</h2><p>Download and extract the knox-{VERSION}.zip}} file into the installation directory that will contain your <a id="\{GATEWAY_HOME\"></a>{GATEWAY_HOME}. You can find the downloads for Knox releases on the [Apache mirrors|http://www.apache.org/dyn/closer.cgi/incubator/knox/].</p>
+</ul><h2><a id="Introduction"></a>Introduction</h2><p>TODO</p><h2><a id="Requirements"></a>Requirements</h2><h3>Java</h3><p>Java 1.6 or later is required for the Knox Gateway runtime. Use the command below to check the version of Java installed on the system where Knox will be running.</p>
+<pre><code>java -version
+</code></pre><h3>Hadoop</h3><p>An an existing Hadoop 1.x or 2.x cluster is required for Knox to protect. One of the easiest ways to ensure this it to utilize a HDP Sandbox VM. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration. Currently if this Hadoop cluster is secured with Kerberos only WebHDFS will work and additional configuration is required.</p><p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow assume that the Gateway is <em>not</em> collocated with the Hadoop clusters themselves and (most importantly) that the hostnames and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running. All of the instructions and samples are tailored to work “out of the
box” against a Hortonworks Sandbox 2.x VM.</p><p>This release of the Apache Knox Gateway has been tested against the <a href="http://hortonworks.com/products/hortonworks-sandbox/">Hortonworks Sandbox 2.0</a>.</p><h2><a id="Download"></a>Download</h2><p>Download and extract the knox-{VERSION}.zip}} file into the installation directory that will contain your <a id="\{GATEWAY_HOME\"></a>{GATEWAY_HOME}. You can find the downloads for Knox releases on the [Apache mirrors|http://www.apache.org/dyn/closer.cgi/incubator/knox/].</p>
<ul>
<li>Source archive: <a href="http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip">knox-incubating-0.3.0-src.zip</a> (<a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-0.3.0-incubating-src.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.md5">MD5 digest</a>)</li>
<li>Binary archive: <a href="http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0.zip">knox-incubating-0.3.0.zip</a> (<a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.md5">MD5 digest</a>)</li>
@@ -155,7 +158,14 @@ Server: Jetty(6.1.26)
<td><img src="question.png" alt="?"/> </td>
</tr>
</tbody>
-</table><h2><a id="Usage+Examples"></a>Usage Examples</h2><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
+</table><h2><a id="Sandbox+Configuration"></a>Sandbox Configuration</h2><p>This version of the Apache Knox Gateway is tested against [Hortonworks Sandbox 1.2|http://hortonworks.com/products/hortonworks-sandbox/]</p><p>Currently there is an issue with Sandbox that prevents it from being easily used with the gateway. In order to correct the issue, you can use the commands below to login to the Sandbox VM and modify the configuration. This assumes that the name sandbox is setup to resolve to the Sandbox VM. It may be necessary to use the IP address of the Sandbox VM instead. <em>This is frequently but not always</em> <a id="{*}192.168.56.101{*"></a>{*}192.168.56.101{*}*.*</p>
+<pre><code>ssh root@sandbox
+cp /usr/lib/hadoop/conf/hdfs-site.xml /usr/lib/hadoop/conf/hdfs-site.xml.orig
+sed -e s/localhost/sandbox/ /usr/lib/hadoop/conf/hdfs-site.xml.orig > /usr/lib/hadoop/conf/hdfs-site.xml
+shutdown -r now
+</code></pre><p>In addition to make it very easy to follow along with the samples for the gateway you can configure your local system to resolve the address of the Sandbox by the names <a id="vm"></a>vm and <a id="sandbox"></a>sandbox. The IP address that is shown below should be that of the Sandbox VM as it is known on your system. This will likely, but not always, be <a id="192.168.56.101"></a>192.168.56.101.</p><p>On Linux or Macintosh systems add a line like this to the end of the file <a id="/etc/hosts"></a>/etc/hosts on your local machine, <em>not the Sandbox VM</em>. <em>Note: The character between the <a id="{_}192.168.56.101{_"></a>{</em>}192.168.56.101{_} and <a id="{_}vm{_"></a>{_}vm{_} below is a <em>{_}tab{_}</em> character._</p>
+<pre><code>192.168.56.101 vm sandbox
+</code></pre><p>On Windows systems a similar but different mechanism can be used. On recent versions of windows the file that should be modified is <a id="%systemroot%\system32\drivers\etc\hosts"></a>%systemroot%\system32\drivers\etc\hosts</p><h2><a id="Usage+Examples"></a>Usage Examples</h2><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
<ul>
<li><a href="#WebHDFS+Examples">WebHDFS</a></li>
<li><a href="#WebHCat+Examples">WebHCat/Templeton</a></li>
@@ -164,26 +174,24 @@ Server: Jetty(6.1.26)
<li><a href="#Hive+Examples">Hive</a></li>
</ul><h2><a id="Configuration"></a>Configuration</h2><h3>Enabling logging</h3><p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the rootLogger value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><h3>Management of Security Artifacts</h3><p>There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
<ol>
- <li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.</li>
-</ol>
-<ul>
- <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. The certificate is stored with an alias of gateway-identity.</li>
- <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
-</ul>
-<ol>
- <li>Look for a credential store at <code>conf/security/keystores/__gateway-credentials.jceks</code>. This credential store is used to store secrets/passwords that are used by the gateway. For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.</li>
-</ol>
-<ul>
- <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <code>gateway-identity-passphrase</code>. This is coordinated with the population of the self-signed cert into the identity-store.</li>
- <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
-</ul><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
+ <li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
+ <ul>
+ <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. The certificate is stored with an alias of gateway-identity.</li>
+ <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
+ </ul></li>
+ <li>Look for a credential store at <code>conf/security/keystores/__gateway-credentials.jceks</code>. This credential store is used to store secrets/passwords that are used by the gateway. For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.
+ <ul>
+ <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <code>gateway-identity-passphrase</code>. This is coordinated with the population of the self-signed cert into the identity-store.</li>
+ <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
+ </ul></li>
+</ol><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
<ol>
- <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>conf/security/keystores/sample-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.</li>
-</ol>
-<ul>
- <li>If no credential store is found for the topology being deployed then one is created for it. Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
- <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
-</ul><p>By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.</p>
+ <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>conf/security/keystores/sample-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.
+ <ul>
+ <li>If no credential store is found for the topology being deployed then one is created for it. Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
+ <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
+ </ul></li>
+</ol><p>By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.</p>
<ol>
<li>Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location and then replicated across all of the slave instances before startup.</li>
<li>Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to replicate them over the network. Of course, NFS mounts have their own challenges.</li>
@@ -192,29 +200,27 @@ Server: Jetty(6.1.26)
<li>Master secret - the same for all gateway instances in a cluster of gateways</li>
<li>All security related artifacts are protected with the master secret</li>
<li>Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways</li>
- <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
+ <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h2><a id="Gateway+Details"></a>Gateway Details</h2><p>TODO</p><h2><a id="Configuration"></a>Configuration</h2><h3>Enabling logging</h3><p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the rootLogger value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><h3>Management of Security Artifacts</h3><p>There are a number of artifacts that are used by the gateway in ens
uring the security of wire level communications, access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
<ol>
- <li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.</li>
-</ol>
-<ul>
- <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. The certificate is stored with an alias of gateway-identity.</li>
- <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
-</ul>
-<ol>
- <li>Look for a credential store at <code>conf/security/keystores/__gateway-credentials.jceks</code>. This credential store is used to store secrets/passwords that are used by the gateway. For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.</li>
-</ol>
-<ul>
- <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <code>gateway-identity-passphrase</code>. This is coordinated with the population of the self-signed cert into the identity-store.</li>
- <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
-</ul><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
+ <li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
+ <ul>
+ <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. The certificate is stored with an alias of gateway-identity.</li>
+ <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
+ </ul></li>
+ <li>Look for a credential store at <code>conf/security/keystores/__gateway-credentials.jceks</code>. This credential store is used to store secrets/passwords that are used by the gateway. For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.
+ <ul>
+ <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <code>gateway-identity-passphrase</code>. This is coordinated with the population of the self-signed cert into the identity-store.</li>
+ <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
+ </ul></li>
+</ol><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
<ol>
- <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>conf/security/keystores/sample-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.</li>
-</ol>
-<ul>
- <li>If no credential store is found for the topology being deployed then one is created for it. Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
- <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
-</ul><p>By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.</p>
+ <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>conf/security/keystores/sample-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.
+ <ul>
+ <li>If no credential store is found for the topology being deployed then one is created for it. Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
+ <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
+ </ul></li>
+</ol><p>By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.</p>
<ol>
<li>Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location and then replicated across all of the slave instances before startup.</li>
<li>Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to replicate them over the network. Of course, NFS mounts have their own challenges.</li>
@@ -223,7 +229,7 @@ Server: Jetty(6.1.26)
<li>Master secret - the same for all gateway instances in a cluster of gateways</li>
<li>All security related artifacts are protected with the master secret</li>
<li>Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways</li>
- <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
+ <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h3><a id="Authentication"></a>Authentication</h3><h4>LDAP Configuration</h4><h4>Session Configuration</h4><h3><a id="Authorization"></a>Authorization</h3><h4>Service Level Authorization</h4><p>The Knox Gateway has an out-of-the-box authorization provider that allows administrators to restrict access to the individual services within a Hadoop cluster.</p><p>This provider utilizes a simple and familiar pattern of using ACLs to protect Hadoop resources by specifying users, groups and ip addresses that are permitted access.</p><p>Note: In the examples below {serviceName} rep
resents a real service name (e.g. WEBHDFS) and would be replaced with these values in an actual configuration.</p><h5>Usecases</h5><h6>USECASE-1: Restrict access to specific Hadoop services to specific Users</h6>
<pre><code><param>
<name>{serviceName}.acl</name>
@@ -475,7 +481,7 @@ Server: Jetty(6.1.26)
<url>http://localhost:10000/</url>
</service>
</topology>
-</code></pre><h2><a id="Client+Details"></a>Client Details</h2><p>Hadoop requires a client that can be used to interact remotely with the services provided by Hadoop cluster. This will also be true when using the Apache Knox Gateway to provide perimeter security and centralized access for these services. The two primary existing clients for Hadoop are the CLI (i.e. Command Line Interface, hadoop) and HUE (i.e. Hadoop User Environment). for several reasons however, neither of these clients can <em>currently</em> be used to access Hadoop services via the Apache Knox Gateway.</p><p>This lead to thinking about a very simple client that could help people use and evaluate the gateway. The list below outline the general requirements for such a client.</p>
+</code></pre><h2><a id="Client+Details"></a>Client Details</h2><p>Hadoop requires a client that can be used to interact remotely with the services provided by Hadoop cluster. This will also be true when using the Apache Knox Gateway to provide perimeter security and centralized access for these services. The two primary existing clients for Hadoop are the CLI (i.e. Command Line Interface, hadoop) and HUE (i.e. Hadoop User Environment). For several reasons however, neither of these clients can <em>currently</em> be used to access Hadoop services via the Apache Knox Gateway.</p><p>This led to thinking about a very simple client that could help people use and evaluate the gateway. The list below outlines the general requirements for such a client.</p>
<ul>
<li>Promote the evaluation and adoption of the Apache Knox Gateway</li>
<li>Simple to deploy and use on data worker desktops to access to remote Hadoop clusters</li>
@@ -485,7 +491,7 @@ Server: Jetty(6.1.26)
<li>Promote the use of REST APIs as the dominant remote client mechanism for Hadoop services</li>
<li>Promote the the sense of Hadoop as a single unified product</li>
<li>Aligned with the Apache Knox Gateway’s overall goals for security</li>
-</ul><p>The result is a very simple DSL (<a href="http://en.wikipedia.org/wiki/Domain-specific_language">Domain Specific Language</a>) of sorts that is used via <a href="http://groovy.codehaus.org">Groovy</a> scripts. Here is an example of a command that copies a file from the local file system to HDFS. <em>Note: The variables session, localFile and remoteFile are assumed to be defined.</em></p>
+</ul><p>The result is a very simple DSL (<a href="http://en.wikipedia.org/wiki/Domain-specific_language">Domain Specific Language</a>) of sorts that is used via <a href="http://groovy.codehaus.org">Groovy</a> scripts. Here is an example of a command that copies a file from the local file system to HDFS.</p><p><em>Note: The variables session, localFile and remoteFile are assumed to be defined.</em></p>
<pre><code>Hdfs.put( session ).file( localFile ).to( remoteFile ).now()
</code></pre><p><em>This work is very early in development but is also very useful in its current state.</em> <em>We are very interested in receiving feedback about how to improve this feature and the DSL in particular.</em></p><p>A note of thanks to <a href="https://code.google.com/p/rest-assured/">REST-assured</a> which provides a <a href="http://en.wikipedia.org/wiki/Fluent_interface">Fluent interface</a> style DSL for testing REST services. It served as the initial inspiration for the creation of this DSL.</p><h3>Assumptions</h3><p>This document assumes a few things about your environment in order to simplify the examples.</p>
<ul>
@@ -1217,7 +1223,684 @@ curl -i -k -u bob:bob-password -X GET \
# 11. Optionally cleanup the test directory
curl -i -k -u bob:bob-password -X DELETE \
'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test?op=DELETE&recursive=true'
-</code></pre><h3><a id="HBase"></a>HBase</h3><p>TODO</p><h4>HBase URL Mapping</h4><p>TODO</p><h4><a id="HBase+Examples"></a>HBase Examples</h4><p>TODO</p><h3><a id="Hive"></a>Hive</h3><p>TODO</p><h4>Hive URL Mapping</h4><p>TODO</p><h4><a id="Hive+Examples"></a>Hive Examples</h4><p>TODO</p><h2><a id="Secure+Clusters"></a>Secure Clusters</h2><p>If your Hadoop cluster is secured with Kerberos authentication, you have to do the following on Knox side.</p><h3>Secure the Hadoop Cluster</h3><p>Please secure Hadoop services with Keberos authentication.</p><p>Please see instructions at [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuration_in_Secure_Mode] and [http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14.html]</p><h3>Create Unix account for Knox on Hadoop master nodes</h3>
+</code></pre><h3><a id="HBase"></a>HBase</h3><p>TODO</p><h4>HBase URL Mapping</h4><p>TODO</p><h4><a id="HBase+Examples"></a>HBase Examples</h4><p>TODO</p><p>The examples below illustrate the set of basic operations with HBase instance using Stargate REST API. Use following link to get more more details about HBase/Stargate API: <a href="http://wiki.apache.org/hadoop/Hbase/Stargate">http://wiki.apache.org/hadoop/Hbase/Stargate</a>.</p><h3>Assumptions</h3><p>This document assumes a few things about your environment in order to simplify the examples.</p>
+<ol>
+ <li>The JVM is executable as simply java.</li>
+ <li>The Apache Knox Gateway is installed and functional.</li>
+ <li>The example commands are executed within the context of the GATEWAY_HOME current directory. The GATEWAY_HOME directory is the directory within the Apache Knox Gateway installation that contains the README file and the bin, conf and deployments directories.</li>
+ <li>A few examples optionally require the use of commands from a standard Groovy installation. These examples are optional but to try them you will need Groovy [installed|http://groovy.codehaus.org/Installing+Groovy].</li>
+</ol><h3>HBase Stargate Setup</h3><h4>Launch Stargate</h4><p>The command below launches the Stargate daemon on port 60080</p>
+<pre><code>sudo /usr/lib/hbase/bin/hbase-daemon.sh start rest -p 60080
+</code></pre><p>60080 post is used because it was specified in sample Hadoop cluster deployment <a id="\{GATEWAY_HOME\"></a>{GATEWAY_HOME}/deployments/sample.xml.</p><h4>Configure Sandbox port mapping for VirtualBox</h4>
+<ol>
+ <li>Select the VM</li>
+ <li>Select menu Machine>Settings…</li>
+ <li>Select tab Network</li>
+ <li>Select Adapter 1</li>
+ <li>Press Port Forwarding button</li>
+ <li>Press Plus button to insert new rule: Name=Stargate, Host Port=60080, Guest Port=60080</li>
+ <li>Press OK to close the rule window</li>
+ <li>Press OK to Network window save the changes</li>
+</ol><p>60080 post is used because it was specified in sample Hadoop cluster deployment <a id="\{GATEWAY_HOME\"></a>{GATEWAY_HOME}/deployments/sample.xml.</p><h3>HBase/Stargate via KnoxShell DSL</h3><h4>Usage</h4><p>For more details about client DSL usage please follow this [page|https://cwiki.apache.org/confluence/display/KNOX/Client+Usage].</p><h5>systemVersion() - Query Software Version.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).systemVersion().now().string"></a>HBase.session(session).systemVersion().now().string</li>
+ </ul></li>
+</ul><h5>clusterVersion() - Query Storage Cluster Version.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).clusterVersion().now().string"></a>HBase.session(session).clusterVersion().now().string</li>
+ </ul></li>
+</ul><h5>status() - Query Storage Cluster Status.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).status().now().string"></a>HBase.session(session).status().now().string</li>
+ </ul></li>
+</ul><h5>table().list() - Query Table List.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example</li>
+ <li><a id="HBase.session(session).table().list().now().string"></a>HBase.session(session).table().list().now().string</li>
+</ul><h5>table(String tableName).schema() - Query Table Schema.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table().schema().now().string"></a>HBase.session(session).table().schema().now().string</li>
+ </ul></li>
+</ul><h5>table(String tableName).create() - Create Table Schema.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>attribute(String name, Object value) - the table’s attribute.</li>
+ <li>family(String name) - starts family definition. Has sub requests:</li>
+ <li>attribute(String name, Object value) - the family’s attribute.</li>
+ <li>endFamilyDef() - finishes family definition.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).create()"></a>HBase.session(session).table(tableName).create() <a id=“.attribute(”tb_attr1“,+”value1“)”></a>.attribute(“tb_attr1”, “value1”) <a id=“.attribute(”tb_attr2“,+”value2“)”></a>.attribute(“tb_attr2”, “value2”) <a id=“.family(”family1“)”></a>.family(“family1”) <a id=“.attribute(”fm_attr1“,+”value3“)”></a>.attribute(“fm_attr1”, “value3”) <a id=“.attribute(”fm_attr2“,+”value4“)”></a>.attribute(“fm_attr2”, “value4”) <a id=".endFamilyDef()"></a>.endFamilyDef() <a id=“.family(”family2“)”></a>.family(“family2”) <a id=“.family(”family3“)”></a>.famil
y(“family3”) <a id=".endFamilyDef()"></a>.endFamilyDef() <a id=“.attribute(”tb_attr3“,+”value5“)”></a>.attribute(“tb_attr3”, “value5”) <a id=".now()"></a>.now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).update() - Update Table Schema.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>family(String name) - starts family definition. Has sub requests:</li>
+ <li>attribute(String name, Object value) - the family’s attribute.</li>
+ <li>endFamilyDef() - finishes family definition.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).update()"></a>HBase.session(session).table(tableName).update() <a id=“.family(”family1“)”></a>.family(“family1”) <a id=“.attribute(”fm_attr1“,+”new_value3“)”></a>.attribute(“fm_attr1”, “new_value3”) <a id=".endFamilyDef()"></a>.endFamilyDef() <a id=“.family(”family4“)”></a>.family(“family4”) <a id=“.attribute(”fm_attr3“,+”value6“)”></a>.attribute(“fm_attr3”, “value6”) <a id=".endFamilyDef()"></a>.endFamilyDef() <a id=".now()"></a>.now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).regions() - Query Table Metadata.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).regions().now().string"></a>HBase.session(session).table(tableName).regions().now().string</li>
+ </ul></li>
+</ul><h5>table(String tableName).delete() - Delete Table.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).delete().now()"></a>HBase.session(session).table(tableName).delete().now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).row(String rowId).store() - Cell Store.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>column(String family, String qualifier, Object value, Long time) - the data to store; “qualifier” may be “null”; “time” is optional.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id=“HBase.session(session).table(tableName).row(”row_id_1“).store()”></a>HBase.session(session).table(tableName).row(“row_id_1”).store() <a id=“.column(”family1“,+”col1“,+”col_value1“)”></a>.column(“family1”, “col1”, “col_value1”) <a id=“.column(”family1“,+”col2“,+”col_value2“,+1234567890l)”></a>.column(“family1”, “col2”, “col_value2”, 1234567890l) <a id=“.column(”family2“,+null,+”fam_value1“)”></a>.column(“family2”, null, “fam_value1”) <a id=".now()"></a>.now()</li>
+ <li><a id=“HBase.session(session).table(tableName).row(”row_id_2“).store()”></a>HBase.session(session).table(tableName).row(“row_id_2”).store() <a id=“.column(”family1“,+”row2_col1“,+”row2_col_value1“)”></a>.column(“family1”, “row2_col1”, “row2_col_value1”) <a id=".now()"></a>.now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).row(String rowId).query() - Cell or Row Query.</h5>
+<ul>
+ <li>rowId is optional. Querying with null or empty rowId will select all rows.</li>
+ <li>Request
+ <ul>
+ <li>column(String family, String qualifier) - the column to select; “qualifier” is optional.</li>
+ <li>startTime(Long) - the lower bound for filtration by time.</li>
+ <li>endTime(Long) - the upper bound for filtration by time.</li>
+ <li>times(Long startTime, Long endTime) - the lower and upper bounds for filtration by time.</li>
+ <li>numVersions(Long) - the maximum number of versions to return.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id=“HBase.session(session).table(tableName).row(”row_id_1“)”></a>HBase.session(session).table(tableName).row(“row_id_1”) <a id=".query()"></a>.query() <a id=".now().string"></a>.now().string</li>
+ <li><a id="HBase.session(session).table(tableName).row().query().now().string"></a>HBase.session(session).table(tableName).row().query().now().string</li>
+ <li><a id="HBase.session(session).table(tableName).row().query()"></a>HBase.session(session).table(tableName).row().query() <a id=“.column(”family1“,+”row2_col1“)”></a>.column(“family1”, “row2_col1”) <a id=“.column(”family2“)”></a>.column(“family2”) <a id=".times(0,+Long.MAX_VALUE)"></a>.times(0, Long.MAX_VALUE) <a id=".numVersions(1)"></a>.numVersions(1) <a id=".now().string"></a>.now().string</li>
+ </ul></li>
+</ul><h5>table(String tableName).row(String rowId).delete() - Row, Column, or Cell Delete.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>column(String family, String qualifier) - the column to delete; “qualifier” is optional.</li>
+ <li>time(Long) - the upper bound for time filtration.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id=“HBase.session(session).table(tableName).row(”row_id_1“)”></a>HBase.session(session).table(tableName).row(“row_id_1”) <a id=".delete()"></a>.delete() <a id=“.column(”family1“,+”col1“)”></a>.column(“family1”, “col1”) <a id=".now()"></a>.now()</li>
+ <li><a id=“HBase.session(session).table(tableName).row(”row_id_1“)”></a>HBase.session(session).table(tableName).row(“row_id_1”) <a id=".delete()"></a>.delete() <a id=“.column(”family2“)”></a>.column(“family2”) <a id=".time(Long.MAX_VALUE)"></a>.time(Long.MAX_VALUE) <a id=".now()"></a>.now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).scanner().create() - Scanner Creation.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>startRow(String) - the lower bound for filtration by row id.</li>
+ <li>endRow(String) - the upper bound for filtration by row id.</li>
+ <li>rows(String startRow, String endRow) - the lower and upper bounds for filtration by row id.</li>
+ <li>column(String family, String qualifier) - the column to select; “qualifier” is optional.</li>
+ <li>batch(Integer) - the batch size.</li>
+ <li>startTime(Long) - the lower bound for filtration by time.</li>
+ <li>endTime(Long) - the upper bound for filtration by time.</li>
+ <li>times(Long startTime, Long endTime) - the lower and upper bounds for filtration by time.</li>
+ <li>filter(String) - the filter XML definition.</li>
+ <li>maxVersions(Integer) - the the maximum number of versions to return.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>scannerId : String - the scanner ID of the created scanner. Consumes body.</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).scanner().create()"></a>HBase.session(session).table(tableName).scanner().create() <a id=“.column(”family1“,+”col2“)”></a>.column(“family1”, “col2”) <a id=“.column(”family2“)”></a>.column(“family2”) <a id=“.startRow(”row_id_1“)”></a>.startRow(“row_id_1”) <a id=“.endRow(”row_id_2“)”></a>.endRow(“row_id_2”) <a id=".batch(1)"></a>.batch(1) <a id=".startTime(0)"></a>.startTime(0) <a id=".endTime(Long.MAX_VALUE)"></a>.endTime(Long.MAX_VALUE) <a id=“.filter(”“)”></a>.filter("") <a id=".maxVersions(100)"></a>.maxVersions(100) <a id=".now()"></a>.now()</li>
+ </ul></li>
+</ul><h5>table(String tableName).scanner(String scannerId).getNext() - Scanner Get Next.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>BasicResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).scanner(scannerId).getNext().now().string"></a>HBase.session(session).table(tableName).scanner(scannerId).getNext().now().string</li>
+ </ul></li>
+</ul><h5>table(String tableName).scanner(String scannerId).delete() - Scanner Deletion.</h5>
+<ul>
+ <li>Request
+ <ul>
+ <li>No request parameters.</li>
+ </ul></li>
+ <li>Response
+ <ul>
+ <li>EmptyResponse</li>
+ </ul></li>
+ <li>Example
+ <ul>
+ <li><a id="HBase.session(session).table(tableName).scanner(scannerId).delete().now()"></a>HBase.session(session).table(tableName).scanner(scannerId).delete().now()</li>
+ </ul></li>
+</ul><h4>Examples</h4><p>This example illustrates sequence of all basic HBase operations: 1. get system version 2. get cluster version 3. get cluster status 4. create the table 5. get list of tables 6. get table schema 7. update table schema 8. insert single row into table 9. query row by id 10. query all rows 11. delete cell from row 12. delete entire column family from row 13. get table regions 14. create scanner 15. fetch values using scanner 16. drop scanner 17. drop the table</p><p>There are several ways to do this depending upon your preference.</p><p>You can use the Groovy interpreter provided with the distribution.</p>
+<pre><code>java -jar bin/shell.jar samples/ExampleHBaseUseCase.groovy
+</code></pre><p>You can manually type in the KnoxShell DSL script into the interactive Groovy interpreter provided with the distribution.</p>
+<pre><code>java -jar bin/shell.jar
+</code></pre><p>Each line from the file below will need to be typed or copied into the interactive shell.</p><p>{code:title=“samples/ExampleHBaseUseCase.groovy”}</p>
+<pre><code>/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.gateway.shell.hbase
+
+import org.apache.hadoop.gateway.shell.Hadoop
+
+import static java.util.concurrent.TimeUnit.SECONDS
+
+gateway = "https://localhost:8443/gateway/sandbox"
+username = "guest"
+password = "guest-password"
+tableName = "test_table"
+
+session = Hadoop.login(gateway, username, password)
+
+println "System version : " + HBase.session(session).systemVersion().now().string
+
+println "Cluster version : " + HBase.session(session).clusterVersion().now().string
+
+println "Status : " + HBase.session(session).status().now().string
+
+println "Creating table '" + tableName + "'..."
+
+HBase.session(session).table(tableName).create() \
+ .attribute("tb_attr1", "value1") \
+ .attribute("tb_attr2", "value2") \
+ .family("family1") \
+ .attribute("fm_attr1", "value3") \
+ .attribute("fm_attr2", "value4") \
+ .endFamilyDef() \
+ .family("family2") \
+ .family("family3") \
+ .endFamilyDef() \
+ .attribute("tb_attr3", "value5") \
+ .now()
+
+println "Done"
+
+println "Table List : " + HBase.session(session).table().list().now().string
+
+println "Schema for table '" + tableName + "' : " + HBase.session(session) \
+ .table(tableName) \
+ .schema() \
+ .now().string
+
+println "Updating schema of table '" + tableName + "'..."
+
+HBase.session(session).table(tableName).update() \
+ .family("family1") \
+ .attribute("fm_attr1", "new_value3") \
+ .endFamilyDef() \
+ .family("family4") \
+ .attribute("fm_attr3", "value6") \
+ .endFamilyDef() \
+ .now()
+
+println "Done"
+
+println "Schema for table '" + tableName + "' : " + HBase.session(session) \
+ .table(tableName) \
+ .schema() \
+ .now().string
+
+println "Inserting data into table..."
+
+HBase.session(session).table(tableName).row("row_id_1").store() \
+ .column("family1", "col1", "col_value1") \
+ .column("family1", "col2", "col_value2", 1234567890l) \
+ .column("family2", null, "fam_value1") \
+ .now()
+
+HBase.session(session).table(tableName).row("row_id_2").store() \
+ .column("family1", "row2_col1", "row2_col_value1") \
+ .now()
+
+println "Done"
+
+println "Querying row by id..."
+
+println HBase.session(session).table(tableName).row("row_id_1") \
+ .query() \
+ .now().string
+
+println "Querying all rows..."
+
+println HBase.session(session).table(tableName).row().query().now().string
+
+println "Querying row by id with extended settings..."
+
+println HBase.session(session).table(tableName).row().query() \
+ .column("family1", "row2_col1") \
+ .column("family2") \
+ .times(0, Long.MAX_VALUE) \
+ .numVersions(1) \
+ .now().string
+
+println "Deleting cell..."
+
+HBase.session(session).table(tableName).row("row_id_1") \
+ .delete() \
+ .column("family1", "col1") \
+ .now()
+
+println "Rows after delete:"
+
+println HBase.session(session).table(tableName).row().query().now().string
+
+println "Extended cell delete"
+
+HBase.session(session).table(tableName).row("row_id_1") \
+ .delete() \
+ .column("family2") \
+ .time(Long.MAX_VALUE) \
+ .now()
+
+println "Rows after delete:"
+
+println HBase.session(session).table(tableName).row().query().now().string
+
+println "Table regions : " + HBase.session(session).table(tableName) \
+ .regions() \
+ .now().string
+
+println "Creating scanner..."
+
+scannerId = HBase.session(session).table(tableName).scanner().create() \
+ .column("family1", "col2") \
+ .column("family2") \
+ .startRow("row_id_1") \
+ .endRow("row_id_2") \
+ .batch(1) \
+ .startTime(0) \
+ .endTime(Long.MAX_VALUE) \
+ .filter("") \
+ .maxVersions(100) \
+ .now().scannerId
+
+println "Scanner id=" + scannerId
+
+println "Scanner get next..."
+
+println HBase.session(session).table(tableName).scanner(scannerId) \
+ .getNext() \
+ .now().string
+
+println "Dropping scanner with id=" + scannerId
+
+HBase.session(session).table(tableName).scanner(scannerId).delete().now()
+
+println "Done"
+
+println "Dropping table '" + tableName + "'..."
+
+HBase.session(session).table(tableName).delete().now()
+
+println "Done"
+
+session.shutdown(10, SECONDS)
+</code></pre><h3>HBase/Stargate via cURL</h3><h4>Get software version</h4><p>Set Accept Header to “text/plain”, “text/xml”, “application/json” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: application/json"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/version'
+</code></pre><h4>Get version information regarding the HBase cluster backing the Stargate instance</h4><p>Set Accept Header to “text/plain”, “text/xml” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/version/cluster'
+</code></pre><h4>Get detailed status on the HBase cluster backing the Stargate instance.</h4><p>Set Accept Header to “text/plain”, “text/xml”, “application/json” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/status/cluster'
+</code></pre><h4>Get the list of available tables.</h4><p>Set Accept Header to “text/plain”, “text/xml”, “application/json” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase'
+</code></pre><h4>Create table with two column families using xml input</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml" -H "Content-Type: text/xml"\
+ -d '<?xml version="1.0" encoding="UTF-8"?><TableSchema name="table1"><ColumnSchema name="family1"/><ColumnSchema name="family2"/></TableSchema>'\
+ -X PUT 'https://localhost:8443/gateway/sandbox/hbase/table1/schema'
+</code></pre><h4>Create table with two column families using JSON input</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: application/json" -H "Content-Type: application/json"\
+ -d '{"name":"table2","ColumnSchema":[{"name":"family3"},{"name":"family4"}]}'\
+ -X PUT 'https://localhost:8443/gateway/sandbox/hbase/table2/schema'
+</code></pre><h4>Get table metadata</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/table1/regions'
+</code></pre><h4>Insert single row table</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Content-Type: text/xml"\
+ -H "Accept: text/xml"\
+ -d '<?xml version="1.0" encoding="UTF-8" standalone="yes"?><CellSet><Row key="cm93MQ=="><Cell column="ZmFtaWx5MTpjb2wx" >dGVzdA==</Cell></Row></CellSet>'\
+ -X POST 'https://localhost:8443/gateway/sandbox/hbase/table1/row1'
+</code></pre><h4>Insert multiple rows into table</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Content-Type: text/xml"\
+ -H "Accept: text/xml"\
+ -d '<?xml version="1.0" encoding="UTF-8" standalone="yes"?><CellSet><Row key="cm93MA=="><Cell column=" ZmFtaWx5Mzpjb2x1bW4x" >dGVzdA==</Cell></Row><Row key="cm93MQ=="><Cell column=" ZmFtaWx5NDpjb2x1bW4x" >dGVzdA==</Cell></Row></CellSet>'\
+ -X POST 'https://localhost:8443/gateway/sandbox/hbase/table2/false-row-key'
+</code></pre><h4>Get all data from table</h4><p>Set Accept Header to “text/plain”, “text/xml”, “application/json” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/table1/*'
+</code></pre><h4>Execute cell or row query</h4><p>Set Accept Header to “text/plain”, “text/xml”, “application/json” or “application/x-protobuf”</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/table1/row1/family1:col1'
+</code></pre><h4>Delete entire row from table</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table2/row0'
+</code></pre><h4>Delete column family from row</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table2/row0/family3'
+</code></pre><h4>Delete specific column from row</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table2/row0/family3'
+</code></pre><h4>Create scanner</h4><p>Scanner URL will be in Location response header</p>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Content-Type: text/xml"\
+ -d '<Scanner batch="1"/>'\
+ -X PUT 'https://localhost:8443/gateway/sandbox/hbase/table1/scanner'
+</code></pre><h4>Get the values of the next cells found by the scanner</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: application/json"\
+ -X GET 'https://localhost:8443/gateway/sandbox/hbase/table1/scanner/13705290446328cff5ed'
+</code></pre><h4>Delete scanner</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -H "Accept: text/xml"\
+ -X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table1/scanner/13705290446328cff5ed'
+</code></pre><h4>Delete table</h4>
+<pre><code>% curl -ik -u guest:guest-password\
+ -X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table1/schema'
+</code></pre><h3><a id="Hive"></a>Hive</h3><p>TODO</p><h4>Hive URL Mapping</h4><p>TODO</p><h4><a id="Hive+Examples"></a>Hive Examples</h4><p>This guide provides detailed examples for how to to some basic interactions with Hive via the Apache Knox Gateway.</p><h5>Assumptions</h5><p>This document assumes a few things about your environment in order to simplify the examples.</p>
+<ol>
+ <li>The JVM is executable as simply java.</li>
+ <li>The Apache Knox Gateway is installed and functional.</li>
+ <li>Minor Hive version is 0.12.0.</li>
+ <li>The example commands are executed within the context of the GATEWAY_HOME current directory. The GATEWAY_HOME directory is the directory within the Apache Knox Gateway installation that contains the README file and the bin, conf and deployments directories.</li>
+ <li>A few examples optionally require the use of commands from a standard Groovy installation. These examples are optional but to try them you will need Groovy [installed|http://groovy.codehaus.org/Installing+Groovy].</li>
+</ol><h5>Setup</h5>
+<ol>
+ <li>Make sure you are running the correct version of Hive to ensure JDBC/Thrift/HTTP support.</li>
+ <li>Make sure Hive is running on the correct port.</li>
+ <li>In hive-server.xml add the property “hive.server2.servermode=http”</li>
+ <li>Client side (JDBC):
+ <ol>
+ <li>Hive JDBC in HTTP mode depends on following libraries to run successfully(must be in the classpath): Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar;</li>
+ <li>import gateway certificate into default truststore. It is located in the <java-home>/lib/security/cacerts: <code>keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore <java-home>/lib/security/cacerts</code></li>
+ <li>connection URL has to be following: <code>jdbc:hive2://<gateway-host>:<gateway-port>/?hive.server2.servermode=https;hive.server2.http.path=<gateway-path>/<cluster-name>/hive</code></li>
+ <li>look at <a href="https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations">https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations</a> for examples; Hint: it would be better to execute “set hive.security.authorization.enabled=false” as the first statement - for testing purposes; <a href="http://gettingstarted.hadooponazure.com/hw/hive.html">http://gettingstarted.hadooponazure.com/hw/hive.html</a> - here is a good example of Hive DDL/DML operations.</li>
+ </ol></li>
+</ol><h5>Customization</h5><p>This example may need to be tailored to the execution environment. In particular host name, host port, user name, user password and context path may need to be changed to match your environment. In particular there is one example file in the distribution that may need to be customized. Take a moment to review this file. All of the values that may need to be customized can be found together at the top of the file.</p>
+<ul>
+ <li>samples/HiveJDBCSample.java</li>
+</ul><h5>Client JDBC Example</h5><p>Sample example for creating new table, loading data into it from local file system and querying data from that table.</p><h6>Java</h6>
+<pre><code>import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.logging.Level;
+import java.util.logging.Logger;
+
+public class HiveJDBCSample {
+
+ public static void main( String[] args ) {
+ Connection connection = null;
+ Statement statement = null;
+ ResultSet resultSet = null;
+
+ try {
+ String user = "guest";
+ String password = user + "-password";
+ String gatewayHost = "localhost";
+ int gatewayPort = 8443;
+ String contextPath = "gateway/sandbox/hive";
+ String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+
+ // load Hive JDBC Driver
+ Class.forName( "org.apache.hive.jdbc.HiveDriver" );
+
+ // configure JDBC connection
+ connection = DriverManager.getConnection( connectionString, user, password );
+
+ statement = connection.createStatement();
+
+ // disable Hive authorization - it could be ommited if Hive authorization
+ // was configured properly
+ statement.execute( "set hive.security.authorization.enabled=false" );
+
+ // create sample table
+ statement.execute( "CREATE TABLE logs(column1 string, column2 string, column3 string, column4 string, column5 string, column6 string, column7 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '" );
+
+ // load data into Hive from file /tmp/log.txt which is placed on the local file system
+ statement.execute( "LOAD DATA LOCAL INPATH '/tmp/log.txt' OVERWRITE INTO TABLE logs" );
+
+ resultSet = statement.executeQuery( "SELECT * FROM logs" );
+
+ while ( resultSet.next() ) {
+ System.out.println( resultSet.getString( 1 ) + " --- " + resultSet.getString( 2 ) + " --- " + resultSet.getString( 3 ) + " --- " + resultSet.getString( 4 ) );
+ }
+ } catch ( ClassNotFoundException ex ) {
+ Logger.getLogger( HiveJDBCSample.class.getName() ).log( Level.SEVERE, null, ex );
+ } catch ( SQLException ex ) {
+ Logger.getLogger( HiveJDBCSample.class.getName() ).log( Level.SEVERE, null, ex );
+ } finally {
+ if ( resultSet != null ) {
+ try {
+ resultSet.close();
+ } catch ( SQLException ex ) {
+ Logger.getLogger( HiveJDBCSample.class.getName() ).log( Level.SEVERE, null, ex );
+ }
+ }
+ if ( statement != null ) {
+ try {
+ statement.close();
+ } catch ( SQLException ex ) {
+ Logger.getLogger( HiveJDBCSample.class.getName() ).log( Level.SEVERE, null, ex );
+ }
+ }
+ if ( connection != null ) {
+ try {
+ connection.close();
+ } catch ( SQLException ex ) {
+ Logger.getLogger( HiveJDBCSample.class.getName() ).log( Level.SEVERE, null, ex );
+ }
+ }
+ }
+ }
+}
+</code></pre><p>h3. Groovy</p><p>Make sure that GATEWAY_HOME/ext directory contains following jars/classes for successful execution: Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar</p><p>There are several ways to execute this sample depending upon your preference.</p><p>You can use the Groovy interpreter provided with the distribution.</p>
+<pre><code>java -jar bin/shell.jar samples/hive/groovy/jdbc/sandbox/HiveJDBCSample.groovy
+</code></pre><p>You can manually type in the KnoxShell DSL script into the interactive Groovy interpreter provided with the distribution.</p>
+<pre><code>java -jar bin/shell.jar
+</code></pre><p>Each line from the file below will need to be typed or copied into the interactive shell.</p>
+<pre><code>import java.sql.DriverManager
+
+user = "guest";
+password = user + "-password";
+gatewayHost = "localhost";
+gatewayPort = 8443;
+contextPath = "gateway/sandbox/hive";
+connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+
+// Load Hive JDBC Driver
+Class.forName( "org.apache.hive.jdbc.HiveDriver" );
+
+// Configure JDBC connection
+connection = DriverManager.getConnection( connectionString, user, password );
+
+statement = connection.createStatement();
+
+// Disable Hive authorization - This can be ommited if Hive authorization is configured properly
+statement.execute( "set hive.security.authorization.enabled=false" );
+
+// Create sample table
+statement.execute( "CREATE TABLE logs(column1 string, column2 string, column3 string, column4 string, column5 string, column6 string, column7 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '" );
+
+// Load data into Hive from file /tmp/log.txt which is placed on the local file system
+statement.execute( "LOAD DATA LOCAL INPATH '/tmp/sample.log' OVERWRITE INTO TABLE logs" );
+
+resultSet = statement.executeQuery( "SELECT * FROM logs" );
+
+while ( resultSet.next() ) {
+ System.out.println( resultSet.getString( 1 ) + " --- " + resultSet.getString( 2 ) );
+}
+
+resultSet.close();
+statement.close();
+connection.close();
+</code></pre><p>Exampes use ‘log.txt’ with content:</p>
+<pre><code>2012-02-03 18:35:34 SampleClass6 [INFO] everything normal for id 577725851
+2012-02-03 18:35:34 SampleClass4 [FATAL] system problem at id 1991281254
+2012-02-03 18:35:34 SampleClass3 [DEBUG] detail for id 1304807656
+2012-02-03 18:35:34 SampleClass3 [WARN] missing id 423340895
+2012-02-03 18:35:34 SampleClass5 [TRACE] verbose detail for id 2082654978
+2012-02-03 18:35:34 SampleClass0 [ERROR] incorrect id 1886438513
+2012-02-03 18:35:34 SampleClass9 [TRACE] verbose detail for id 438634209
+2012-02-03 18:35:34 SampleClass8 [DEBUG] detail for id 2074121310
+2012-02-03 18:35:34 SampleClass0 [TRACE] verbose detail for id 1505582508
+2012-02-03 18:35:34 SampleClass0 [TRACE] verbose detail for id 1903854437
+2012-02-03 18:35:34 SampleClass7 [DEBUG] detail for id 915853141
+2012-02-03 18:35:34 SampleClass3 [TRACE] verbose detail for id 303132401
+2012-02-03 18:35:34 SampleClass6 [TRACE] verbose detail for id 151914369
+2012-02-03 18:35:34 SampleClass2 [DEBUG] detail for id 146527742
+...
+</code></pre><p>Expected output:</p>
+<pre><code>2012-02-03 --- 18:35:34 --- SampleClass6 --- [INFO]
+2012-02-03 --- 18:35:34 --- SampleClass4 --- [FATAL]
+2012-02-03 --- 18:35:34 --- SampleClass3 --- [DEBUG]
+2012-02-03 --- 18:35:34 --- SampleClass3 --- [WARN]
+2012-02-03 --- 18:35:34 --- SampleClass5 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass0 --- [ERROR]
+2012-02-03 --- 18:35:34 --- SampleClass9 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass8 --- [DEBUG]
+2012-02-03 --- 18:35:34 --- SampleClass0 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass0 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass7 --- [DEBUG]
+2012-02-03 --- 18:35:34 --- SampleClass3 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass6 --- [TRACE]
+2012-02-03 --- 18:35:34 --- SampleClass2 --- [DEBUG]
+...
+</code></pre><h2><a id="Secure+Clusters"></a>Secure Clusters</h2><p>If your Hadoop cluster is secured with Kerberos authentication, you have to do the following on Knox side.</p><h3>Secure the Hadoop Cluster</h3><p>Please secure Hadoop services with Keberos authentication.</p><p>Please see instructions at [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuration_in_Secure_Mode] and [http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14.html]</p><h3>Create Unix account for Knox on Hadoop master nodes</h3>
<pre><code>useradd \-g hadoop knox
</code></pre><h3>Create Kerberos principal, keytab for Knox</h3><p>One way of doing this, assuming your KDC realm is EXAMPLE.COM</p><p>ssh into your host running KDC</p>
<pre><code>kadmin.local
Modified: incubator/knox/trunk/books/0.3.0/book.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book.md?rev=1526308&r1=1526307&r2=1526308&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book.md (original)
+++ incubator/knox/trunk/books/0.3.0/book.md Wed Sep 25 21:40:30 2013
@@ -29,6 +29,7 @@ Table Of Contents
* [Installation](#Installation)
* [Getting Started](#Getting+Started)
* [Supported Services](#Supported+Services)
+* [Sandbox Configuration](#Sandbox+Configuration)
* [Usage Examples](#Usage+Examples)
* [Gateway Details](#Gateway+Details)
* [Authentication](#Authentication)
@@ -61,9 +62,7 @@ TODO
Java 1.6 or later is required for the Knox Gateway runtime.
Use the command below to check the version of Java installed on the system where Knox will be running.
-{code}
-java -version
-{code}
+ java -version
### Hadoop ###
@@ -128,6 +127,9 @@ Only more recent versions of some Hadoop
| Hive/ODBC | 0.12.0 | ![?] | ![?] |
+<<sandbox.md>>
+
+
{{Usage Examples}}
------------------
Modified: incubator/knox/trunk/books/0.3.0/client.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/client.md?rev=1526308&r1=1526307&r2=1526308&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/client.md (original)
+++ incubator/knox/trunk/books/0.3.0/client.md Wed Sep 25 21:40:30 2013
@@ -21,10 +21,10 @@
Hadoop requires a client that can be used to interact remotely with the services provided by Hadoop cluster.
This will also be true when using the Apache Knox Gateway to provide perimeter security and centralized access for these services.
The two primary existing clients for Hadoop are the CLI (i.e. Command Line Interface, hadoop) and HUE (i.e. Hadoop User Environment).
-for several reasons however, neither of these clients can _currently_ be used to access Hadoop services via the Apache Knox Gateway.
+For several reasons however, neither of these clients can _currently_ be used to access Hadoop services via the Apache Knox Gateway.
-This lead to thinking about a very simple client that could help people use and evaluate the gateway.
-The list below outline the general requirements for such a client.
+This led to thinking about a very simple client that could help people use and evaluate the gateway.
+The list below outlines the general requirements for such a client.
* Promote the evaluation and adoption of the Apache Knox Gateway
* Simple to deploy and use on data worker desktops to access to remote Hadoop clusters
@@ -37,6 +37,7 @@ The list below outline the general requi
The result is a very simple DSL ([Domain Specific Language](http://en.wikipedia.org/wiki/Domain-specific_language)) of sorts that is used via [Groovy](http://groovy.codehaus.org) scripts.
Here is an example of a command that copies a file from the local file system to HDFS.
+
_Note: The variables session, localFile and remoteFile are assumed to be defined._
Hdfs.put( session ).file( localFile ).to( remoteFile ).now()
Modified: incubator/knox/trunk/books/0.3.0/config.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/config.md?rev=1526308&r1=1526307&r2=1526308&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/config.md (original)
+++ incubator/knox/trunk/books/0.3.0/config.md Wed Sep 25 21:40:30 2013
@@ -35,28 +35,24 @@ The following is a description of how th
Upon start of the gateway server we:
1. Look for an identity store at `conf/security/keystores/gateway.jks`.
-The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
-
-* If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode.
-The certificate is stored with an alias of gateway-identity.
-* If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.
-
+ The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
+ * If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode.
+ The certificate is stored with an alias of gateway-identity.
+ * If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.
2. Look for a credential store at `conf/security/keystores/__gateway-credentials.jceks`.
-This credential store is used to store secrets/passwords that are used by the gateway.
-For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.
-
-* If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias `gateway-identity-passphrase`.
-This is coordinated with the population of the self-signed cert into the identity-store.
-* If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.
+ This credential store is used to store secrets/passwords that are used by the gateway.
+ For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.
+ * If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias `gateway-identity-passphrase`.
+ This is coordinated with the population of the self-signed cert into the identity-store.
+ * If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.
Upon deployment of a Hadoop cluster topology within the gateway we:
1. Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for `conf/security/keystores/sample-credentials.jceks`. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.
-
-* If no credential store is found for the topology being deployed then one is created for it.
-Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task.
-They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.
-* If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.
+ * If no credential store is found for the topology being deployed then one is created for it.
+ Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task.
+ They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.
+ * If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.
By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.
@@ -69,8 +65,8 @@ Summary of Secrets to be Managed:
2. All security related artifacts are protected with the master secret
3. Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways
4. Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances.
-However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another.
-This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.
+ However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another.
+ This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.
NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster.
When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue.