You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by mk...@apache.org on 2014/03/03 17:54:44 UTC
svn commit: r1573609 - in /knox: site/ site/books/knox-incubating-0-4-0/
trunk/books/0.4.0/
Author: mkononenko
Date: Mon Mar 3 16:54:43 2014
New Revision: 1573609
URL: http://svn.apache.org/r1573609
Log:
KNOX-287: Updateed domentation to be consistent with Hive 0.12 configuration
Modified:
knox/site/books/knox-incubating-0-4-0/knox-incubating-0-4-0.html
knox/site/index.html
knox/site/issue-tracking.html
knox/site/license.html
knox/site/mail-lists.html
knox/site/project-info.html
knox/site/team-list.html
knox/trunk/books/0.4.0/book_gateway-details.md
knox/trunk/books/0.4.0/config_authz.md
knox/trunk/books/0.4.0/service_hive.md
knox/trunk/books/0.4.0/service_webhcat.md
Modified: knox/site/books/knox-incubating-0-4-0/knox-incubating-0-4-0.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-incubating-0-4-0/knox-incubating-0-4-0.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/books/knox-incubating-0-4-0/knox-incubating-0-4-0.html (original)
+++ knox/site/books/knox-incubating-0-4-0/knox-incubating-0-4-0.html Mon Mar 3 16:54:43 2014
@@ -310,7 +310,12 @@ Server: Jetty(6.1.26)
<li>Gateway: <code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase</code></li>
<li>Cluster: <code>http://{hbase-host}:60080</code></li>
</ul></li>
-</ul><p>The values for <code>{gateway-host}</code>, <code>{gateway-port}</code>, <code>{gateway-path}</code> are provided via the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>).</p><p>The value for <code>{cluster-name}</code> is derived from the file name of the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>The value for <code>{webhdfs-host}</code>, <code>{webhcat-host}</code>, <code>{oozie-host}</code> and <code>{hbase-host}</code> are provided via the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>Note: The ports 50070, 50111, 11000 and 60080 are the defaults for WebHDFS, WebHCat, Oozie and Stargate/HBase respectively. Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.</p><h3><a id="Configuration"></a>Configuration</h3><h4><a id="Topology+Descriptors"></a>Topology Descriptor
s</h4><p>The topology descriptor files provide the gateway with per-cluster configuration information. This includes configuration for both the providers within the gateway and the services within the Hadoop cluster. These files are located in <code>{GATEWAY_HOME}/deployments</code>. The general outline of this document looks like this.</p>
+ <li>Hive JDBC
+ <ul>
+ <li>Gateway: <code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
+ <li>Cluster: <code>http://{hive-host}:10001/cliservice</code></li>
+ </ul></li>
+</ul><p>The values for <code>{gateway-host}</code>, <code>{gateway-port}</code>, <code>{gateway-path}</code> are provided via the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>).</p><p>The value for <code>{cluster-name}</code> is derived from the file name of the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>The value for <code>{webhdfs-host}</code>, <code>{webhcat-host}</code>, <code>{oozie-host}</code>, <code>{hbase-host}</code> and <code>{hive-host}</code> are provided via the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>Note: The ports 50070, 50111, 11000, 60080 and 10001 are the defaults for WebHDFS, WebHCat, Oozie, Stargate/HBase and Hive respectively. Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.</p><h3><a id="Configuration"></a>Configuration</h3><h4><a id="Topolo
gy+Descriptors"></a>Topology Descriptors</h4><p>The topology descriptor files provide the gateway with per-cluster configuration information. This includes configuration for both the providers within the gateway and the services within the Hadoop cluster. These files are located in <code>{GATEWAY_HOME}/deployments</code>. The general outline of this document looks like this.</p>
<pre><code><topology>
<gateway>
<provider>
@@ -985,7 +990,7 @@ ldapRealm.userDnTemplate=uid={0},ou=peop
<service>
<role>HIVE</role>
- <url>http://localhost:10000</url>
+ <url>http://localhost:10001/cliservice</url>
</service>
</topology>
</code></pre><h3><a id="Secure+Clusters"></a>Secure Clusters</h3><p>See these documents for setting up a secure Hadoop cluster <a href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuration_in_Secure_Mode">http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuration_in_Secure_Mode</a> <a href="http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14.html">http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14.html</a></p><p>Once you have a Hadoop cluster that is using Kerberos for authentication, you have to do the following to configure Knox to work with that cluster.</p><h4><a id="Create+Unix+account+for+Knox+on+Hadoop+master+nodes"></a>Create Unix account for Knox on Hadoop master nodes</h4>
@@ -1632,7 +1637,7 @@ session.shutdown()
<name>yarn.resourcemanager.address</name>
<value>sandbox.hortonworks.com:8050</value>
</property>
-</code></pre><p>See <a href="#WebHDFS">WebHDFS</a> for details about locating the Haddop configuration for the NAMENODE endpoint.</p><p>The gateway by default includes a sample topology descriptor file <code>{GATEWAY_HOME}/deployments/sandbox.xml</code>. The values in this sample are configured to work with an installed Sandbox VM.</p>
+</code></pre><p>See <a href="#WebHDFS">WebHDFS</a> for details about locating the Hadoop configuration for the NAMENODE endpoint.</p><p>The gateway by default includes a sample topology descriptor file <code>{GATEWAY_HOME}/deployments/sandbox.xml</code>. The values in this sample are configured to work with an installed Sandbox VM.</p>
<pre><code><service>
<role>NAMENODE</role>
<url>hdfs://localhost:8020</url>
@@ -2421,17 +2426,72 @@ session.shutdown(10, SECONDS)
</code></pre><h4><a id="Delete+table"></a>Delete table</h4>
<pre><code>curl -ik -u guest:guest-password\
-X DELETE 'https://localhost:8443/gateway/sandbox/hbase/table1/schema'
-</code></pre><h3><a id="Hive"></a>Hive</h3><p>TODO</p><h4><a id="Hive+URL+Mapping"></a>Hive URL Mapping</h4><p>TODO</p><h4><a id="Hive+Examples"></a>Hive Examples</h4><p>This guide provides detailed examples for how to to some basic interactions with Hive via the Apache Knox Gateway.</p><h5><a id="Hive+Setup"></a>Hive Setup</h5>
+</code></pre><h3><a id="Hive"></a>Hive</h3><p>The <a href="https://cwiki.apache.org/confluence/display/Hive/Home">Hive wiki pages</a> describe Hive installation and configuration processes. In sandbox configuration file for Hive is located at /etc/hive/hive-site.xml. Hive Server has to be started in HTTP mode. Note the properties shown below as they are related to configuration required by the gateway.</p>
+<pre><code><property>
+ <name>hive.server2.thrift.http.port</name>
+ <value>10001</value>
+ <description>Port number when in HTTP mode.</description>
+</property>
+
+<property>
+ <name>hive.server2.thrift.http.path</name>
+ <value>cliservice</value>
+ <description>Path component of URL endpoint when in HTTP mode.</description>
+</property>
+
+<property>
+ <name>hive.server2.transport.mode</name>
+ <value>http</value>
+ <description>Server transport mode. "binary" or "http".</description>
+</property>
+</code></pre><p>Also current version of Hive doesn’t not support impersonation in HTTP mode, so a following property has to be present in a Hive configuration.</p>
+<pre><code><property>
+ <name>hive.server2.enable.doAs</name>
+ <value>false</value>
+ <description>Set this property to enable impersonation in Hive Server 2</description>
+</property>
+</code></pre><p>The gateway by default includes a sample topology descriptor file <code>{GATEWAY_HOME}/deployments/sandbox.xml</code>. The value in this sample is configured to work with an installed Sandbox VM.</p>
+<pre><code><service>
+ <role>HIVE</role>
+ <url>http://localhost:10001/cliservice</url>
+</service>
+</code></pre><p>By default the gateway is configured to use the binary transport mode for Hive in the Sandbox.</p><h4><a id="Hive+JDBC+URL+Mapping"></a>Hive JDBC URL Mapping</h4>
+<table>
+ <tbody>
+ <tr>
+ <td>Gateway </td>
+ <td><code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code> </td>
+ </tr>
+ <tr>
+ <td>Cluster </td>
+ <td><code>http://{hive-host}:{hive-port}/{hive-path}</code> </td>
+ </tr>
+ </tbody>
+</table><h4><a id="Hive+Examples"></a>Hive Examples</h4><p>This guide provides detailed examples for how to to some basic interactions with Hive via the Apache Knox Gateway.</p><h5><a id="Hive+Setup"></a>Hive Setup</h5>
<ol>
<li>Make sure you are running the correct version of Hive to ensure JDBC/Thrift/HTTP support.</li>
- <li>Make sure Hive is running on the correct port.</li>
- <li>In hive-server.xml add the property “hive.server2.servermode=http”</li>
+ <li>Make sure Hive Server is running on the correct port.</li>
+ <li>Make sure Hive Server is running in HTTP mode.</li>
<li>Client side (JDBC):
<ol>
- <li>Hive JDBC in HTTP mode depends on following libraries to run successfully(must be in the classpath): Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar;</li>
- <li>Import gateway certificate into the default JRE truststore. It is located in the <code>/lib/security/cacerts</code> <code>keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore <java-home>/lib/security/cacerts</code> Alternatively you can run your sample with additional parameters: <code>-Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStore=<path-to-trust-store> -Djavax.net.ssl.trustStorePassword=<trust-store-password></code></li>
- <li>Connection URL has to be following: <code>jdbc:hive2://<gateway-host>:<gateway-port>/?hive.server2.servermode=https;hive.server2.http.path=<gateway-path>/<cluster-name>/hive</code></li>
- <li>Look at <a href="https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations">https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations</a> for examples. Hint: For testing it would be better to execute “set hive.security.authorization.enabled=false” as the first statement. Hint: Good examples of Hive DDL/DML can be found here <a href="http://gettingstarted.hadooponazure.com/hw/hive.html">http://gettingstarted.hadooponazure.com/hw/hive.html</a></li>
+ <li>Hive JDBC in HTTP mode depends on following libraries to run successfully(must be in the classpath):
+ <ul>
+ <li>hadoop-common-2.2.0.2.0.6.0-76.jar;</li>
+ <li>hive-jdbc-0.12.0.2.0.6.0-76.jar;</li>
+ <li>hive-service-0.12.0.2.0.6.0-76.jar;</li>
+ <li>libthrift-0.9.0.jar;</li>
+ <li>httpcore-4.1.4.jar;</li>
+ <li>httpclient-4.1.3.jar;</li>
+ <li>hive-common-0.12.0.2.0.6.0-76.jar;</li>
+ <li>commons-logging-1.1.1.jar;</li>
+ <li>slf4j-api-1.7.5.jar;</li>
+ <li>slf4j-log4j12-1.7.5.jar;</li>
+ <li>log4j-1.2.17.jar;</li>
+ <li>commons-codec-1.7.jar;</li>
+ </ul></li>
+ <li>Import gateway certificate into the default JRE truststore. It is located in the <code>/lib/security/cacerts</code>. <code>keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore <java-home>/lib/security/cacerts</code> Alternatively you can run your sample with additional parameters: <code>-Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStore=<path-to-trust-store> -Djavax.net.ssl.trustStorePassword=<trust-store-password></code></li>
+ <li>Connection URL has to be following: <code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
+ <li>Look at <a href="https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations">https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations</a> for examples. Hint: For testing it would be better to execute <code>set hive.security.authorization.enabled=false</code> as the first statement. Hint: Good examples of Hive DDL/DML can be found here <a href="http://gettingstarted.hadooponazure.com/hw/hive.html">http://gettingstarted.hadooponazure.com/hw/hive.html</a></li>
</ol></li>
</ol><h5><a id="Customization"></a>Customization</h5><p>This example may need to be tailored to the execution environment. In particular host name, host port, user name, user password and context path may need to be changed to match your environment. In particular there is one example file in the distribution that may need to be customized. Take a moment to review this file. All of the values that may need to be customized can be found together at the top of the file.</p>
<ul>
@@ -2459,7 +2519,7 @@ public class HiveJDBCSample {
String gatewayHost = "localhost";
int gatewayPort = 8443;
String contextPath = "gateway/sandbox/hive";
- String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+ String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s", gatewayHost, gatewayPort, contextPath );
// load Hive JDBC Driver
Class.forName( "org.apache.hive.jdbc.HiveDriver" );
@@ -2513,7 +2573,21 @@ public class HiveJDBCSample {
}
}
}
-</code></pre><p>h3. Groovy</p><p>Make sure that GATEWAY_HOME/ext directory contains following jars/classes for successful execution: Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar</p><p>There are several ways to execute this sample depending upon your preference.</p><p>You can use the Groovy interpreter provided with the distribution.</p>
+</code></pre><h6><a id="Groovy"></a>Groovy</h6><p>Make sure that GATEWAY_HOME/ext directory contains following libraries for successful execution:</p>
+<ul>
+ <li>hadoop-common-2.2.0.2.0.6.0-76.jar;</li>
+ <li>hive-jdbc-0.12.0.2.0.6.0-76.jar;</li>
+ <li>hive-service-0.12.0.2.0.6.0-76.jar;</li>
+ <li>libthrift-0.9.0.jar;</li>
+ <li>httpcore-4.1.4.jar;</li>
+ <li>httpclient-4.1.3.jar;</li>
+ <li>hive-common-0.12.0.2.0.6.0-76.jar;</li>
+ <li>commons-logging-1.1.1.jar;</li>
+ <li>slf4j-api-1.7.5.jar;</li>
+ <li>slf4j-log4j12-1.7.5.jar;</li>
+ <li>log4j-1.2.17.jar;</li>
+ <li>commons-codec-1.7.jar;</li>
+</ul><p>There are several ways to execute this sample depending upon your preference.</p><p>You can use the Groovy interpreter provided with the distribution.</p>
<pre><code>java -jar bin/shell.jar samples/hive/groovy/jdbc/sandbox/HiveJDBCSample.groovy
</code></pre><p>You can manually type in the KnoxShell DSL script into the interactive Groovy interpreter provided with the distribution.</p>
<pre><code>java -jar bin/shell.jar
@@ -2525,7 +2599,7 @@ password = user + "-password";
gatewayHost = "localhost";
gatewayPort = 8443;
contextPath = "gateway/sandbox/hive";
-connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s", gatewayHost, gatewayPort, contextPath );
// Load Hive JDBC Driver
Class.forName( "org.apache.hive.jdbc.HiveDriver" );
Modified: knox/site/index.html
URL: http://svn.apache.org/viewvc/knox/site/index.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/index.html (original)
+++ knox/site/index.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/knox/site/issue-tracking.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/issue-tracking.html (original)
+++ knox/site/issue-tracking.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/license.html
URL: http://svn.apache.org/viewvc/knox/site/license.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/license.html (original)
+++ knox/site/license.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/knox/site/mail-lists.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/mail-lists.html (original)
+++ knox/site/mail-lists.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/project-info.html
URL: http://svn.apache.org/viewvc/knox/site/project-info.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/project-info.html (original)
+++ knox/site/project-info.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/site/team-list.html
URL: http://svn.apache.org/viewvc/knox/site/team-list.html?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/site/team-list.html (original)
+++ knox/site/team-list.html Mon Mar 3 16:54:43 2014
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 19, 2014 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Feb 28, 2014 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
- <meta name="Date-Revision-yyyymmdd" content="20140219" />
+ <meta name="Date-Revision-yyyymmdd" content="20140228" />
<meta http-equiv="Content-Language" content="en" />
<script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
<a href="https://cwiki.apache.org/confluence/display/KNOX/Index" class="externalLink" title="Wiki">Wiki</a>
- | <span id="publishDate">Last Published: 2014-02-19</span>
+ | <span id="publishDate">Last Published: 2014-02-28</span>
| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
</div>
<div class="clear">
Modified: knox/trunk/books/0.4.0/book_gateway-details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.4.0/book_gateway-details.md?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/trunk/books/0.4.0/book_gateway-details.md (original)
+++ knox/trunk/books/0.4.0/book_gateway-details.md Mon Mar 3 16:54:43 2014
@@ -40,14 +40,17 @@ The actual port number may be different
* Stargate (HBase)
* Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase`
* Cluster: `http://{hbase-host}:60080`
+* Hive JDBC
+ * Gateway: `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
+ * Cluster: `http://{hive-host}:10001/cliservice`
The values for `{gateway-host}`, `{gateway-port}`, `{gateway-path}` are provided via the gateway configuration file (i.e. `{GATEWAY_HOME}/conf/gateway-site.xml`).
The value for `{cluster-name}` is derived from the file name of the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
-The value for `{webhdfs-host}`, `{webhcat-host}`, `{oozie-host}` and `{hbase-host}` are provided via the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
+The value for `{webhdfs-host}`, `{webhcat-host}`, `{oozie-host}`, `{hbase-host}` and `{hive-host}` are provided via the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
-Note: The ports 50070, 50111, 11000 and 60080 are the defaults for WebHDFS, WebHCat, Oozie and Stargate/HBase respectively.
+Note: The ports 50070, 50111, 11000, 60080 and 10001 are the defaults for WebHDFS, WebHCat, Oozie, Stargate/HBase and Hive respectively.
Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.
<<config.md>>
Modified: knox/trunk/books/0.4.0/config_authz.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.4.0/config_authz.md?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/trunk/books/0.4.0/config_authz.md (original)
+++ knox/trunk/books/0.4.0/config_authz.md Mon Mar 3 16:54:43 2014
@@ -344,6 +344,6 @@ An example of a full topology that illus
<service>
<role>HIVE</role>
- <url>http://localhost:10000</url>
+ <url>http://localhost:10001/cliservice</url>
</service>
</topology>
Modified: knox/trunk/books/0.4.0/service_hive.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.4.0/service_hive.md?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/trunk/books/0.4.0/service_hive.md (original)
+++ knox/trunk/books/0.4.0/service_hive.md Mon Mar 3 16:54:43 2014
@@ -17,11 +17,52 @@
### Hive ###
-TODO
-
-#### Hive URL Mapping ####
-
-TODO
+The [Hive wiki pages](https://cwiki.apache.org/confluence/display/Hive/Home) describe Hive installation and configuration processes.
+In sandbox configuration file for Hive is located at /etc/hive/hive-site.xml.
+Hive Server has to be started in HTTP mode.
+Note the properties shown below as they are related to configuration required by the gateway.
+
+ <property>
+ <name>hive.server2.thrift.http.port</name>
+ <value>10001</value>
+ <description>Port number when in HTTP mode.</description>
+ </property>
+
+ <property>
+ <name>hive.server2.thrift.http.path</name>
+ <value>cliservice</value>
+ <description>Path component of URL endpoint when in HTTP mode.</description>
+ </property>
+
+ <property>
+ <name>hive.server2.transport.mode</name>
+ <value>http</value>
+ <description>Server transport mode. "binary" or "http".</description>
+ </property>
+
+Also current version of Hive doesn't not support impersonation in HTTP mode, so a following property has to be present in a Hive configuration.
+
+ <property>
+ <name>hive.server2.enable.doAs</name>
+ <value>false</value>
+ <description>Set this property to enable impersonation in Hive Server 2</description>
+ </property>
+
+The gateway by default includes a sample topology descriptor file `{GATEWAY_HOME}/deployments/sandbox.xml`.
+The value in this sample is configured to work with an installed Sandbox VM.
+
+ <service>
+ <role>HIVE</role>
+ <url>http://localhost:10001/cliservice</url>
+ </service>
+
+By default the gateway is configured to use the binary transport mode for Hive in the Sandbox.
+
+#### Hive JDBC URL Mapping ####
+
+| ------- | ------------------------------------------------------------------------------- |
+| Gateway | `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive` |
+| Cluster | `http://{hive-host}:{hive-port}/{hive-path}` |
#### Hive Examples ####
@@ -30,20 +71,31 @@ This guide provides detailed examples fo
##### Hive Setup #####
1. Make sure you are running the correct version of Hive to ensure JDBC/Thrift/HTTP support.
-2. Make sure Hive is running on the correct port.
-3. In hive-server.xml add the property "hive.server2.servermode=http"
+2. Make sure Hive Server is running on the correct port.
+3. Make sure Hive Server is running in HTTP mode.
4. Client side (JDBC):
1. Hive JDBC in HTTP mode depends on following libraries to run successfully(must be in the classpath):
- Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar;
+ * hadoop-common-2.2.0.2.0.6.0-76.jar;
+ * hive-jdbc-0.12.0.2.0.6.0-76.jar;
+ * hive-service-0.12.0.2.0.6.0-76.jar;
+ * libthrift-0.9.0.jar;
+ * httpcore-4.1.4.jar;
+ * httpclient-4.1.3.jar;
+ * hive-common-0.12.0.2.0.6.0-76.jar;
+ * commons-logging-1.1.1.jar;
+ * slf4j-api-1.7.5.jar;
+ * slf4j-log4j12-1.7.5.jar;
+ * log4j-1.2.17.jar;
+ * commons-codec-1.7.jar;
2. Import gateway certificate into the default JRE truststore.
- It is located in the `/lib/security/cacerts`
+ It is located in the `/lib/security/cacerts`.
`keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore <java-home>/lib/security/cacerts`
Alternatively you can run your sample with additional parameters:
`-Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStore=<path-to-trust-store> -Djavax.net.ssl.trustStorePassword=<trust-store-password>`
3. Connection URL has to be following:
- `jdbc:hive2://<gateway-host>:<gateway-port>/?hive.server2.servermode=https;hive.server2.http.path=<gateway-path>/<cluster-name>/hive`
+ `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
4. Look at https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations for examples.
- Hint: For testing it would be better to execute "set hive.security.authorization.enabled=false" as the first statement.
+ Hint: For testing it would be better to execute `set hive.security.authorization.enabled=false` as the first statement.
Hint: Good examples of Hive DDL/DML can be found here http://gettingstarted.hadooponazure.com/hw/hive.html
##### Customization #####
@@ -84,7 +136,7 @@ Sample example for creating new table, l
String gatewayHost = "localhost";
int gatewayPort = 8443;
String contextPath = "gateway/sandbox/hive";
- String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+ String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s", gatewayHost, gatewayPort, contextPath );
// load Hive JDBC Driver
Class.forName( "org.apache.hive.jdbc.HiveDriver" );
@@ -139,10 +191,22 @@ Sample example for creating new table, l
}
}
-h3. Groovy
+###### Groovy ######
+
+Make sure that GATEWAY_HOME/ext directory contains following libraries for successful execution:
-Make sure that GATEWAY_HOME/ext directory contains following jars/classes for successful execution:
-Hive Thrift artifacts classes, commons-codec.jar, commons-configuration.jar, commons-lang.jar, commons-logging.jar, hadoop-core.jar, hive-cli.jar, hive-common.jar, hive-jdbc.jar, hive-service.jar, hive-shims.jar, httpclient.jar, httpcore.jar, slf4j-api.jar
+- hadoop-common-2.2.0.2.0.6.0-76.jar;
+- hive-jdbc-0.12.0.2.0.6.0-76.jar;
+- hive-service-0.12.0.2.0.6.0-76.jar;
+- libthrift-0.9.0.jar;
+- httpcore-4.1.4.jar;
+- httpclient-4.1.3.jar;
+- hive-common-0.12.0.2.0.6.0-76.jar;
+- commons-logging-1.1.1.jar;
+- slf4j-api-1.7.5.jar;
+- slf4j-log4j12-1.7.5.jar;
+- log4j-1.2.17.jar;
+- commons-codec-1.7.jar;
There are several ways to execute this sample depending upon your preference.
@@ -163,7 +227,7 @@ Each line from the file below will need
gatewayHost = "localhost";
gatewayPort = 8443;
contextPath = "gateway/sandbox/hive";
- connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.servermode=https;hive.server2.http.path=%s", gatewayHost, gatewayPort, contextPath );
+ connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s", gatewayHost, gatewayPort, contextPath );
// Load Hive JDBC Driver
Class.forName( "org.apache.hive.jdbc.HiveDriver" );
Modified: knox/trunk/books/0.4.0/service_webhcat.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.4.0/service_webhcat.md?rev=1573609&r1=1573608&r2=1573609&view=diff
==============================================================================
--- knox/trunk/books/0.4.0/service_webhcat.md (original)
+++ knox/trunk/books/0.4.0/service_webhcat.md Mon Mar 3 16:54:43 2014
@@ -38,7 +38,7 @@ The property yarn.resourcemanager.addres
<value>sandbox.hortonworks.com:8050</value>
</property>
-See #[WebHDFS] for details about locating the Haddop configuration for the NAMENODE endpoint.
+See #[WebHDFS] for details about locating the Hadoop configuration for the NAMENODE endpoint.
The gateway by default includes a sample topology descriptor file `{GATEWAY_HOME}/deployments/sandbox.xml`.
The values in this sample are configured to work with an installed Sandbox VM.