You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@phoenix.apache.org by st...@apache.org on 2023/12/01 17:02:03 UTC
svn commit: r1914267 - in /phoenix/site: publish/ publish/language/ source/src/site/markdown/
Author: stoty
Date: Fri Dec 1 17:02:03 2023
New Revision: 1914267
URL: http://svn.apache.org/viewvc?rev=1914267&view=rev
Log:
Update Hive connector page, and document webpage building quirks
Modified:
phoenix/site/publish/building_website.html
phoenix/site/publish/classpath_and_url.html
phoenix/site/publish/hive_storage_handler.html
phoenix/site/publish/language/datatypes.html
phoenix/site/publish/language/functions.html
phoenix/site/publish/language/index.html
phoenix/site/source/src/site/markdown/building_website.md
phoenix/site/source/src/site/markdown/hive_storage_handler.md
Modified: phoenix/site/publish/building_website.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/building_website.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/building_website.html (original)
+++ phoenix/site/publish/building_website.html Fri Dec 1 17:02:03 2023
@@ -1,14 +1,14 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
- <title>Building Phoenix Project Web Site | Apache Phoenix</title>
+ <title>Prerequisites | Apache Phoenix</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="" />
<meta http-equiv="content-language" content="en" />
@@ -168,8 +168,14 @@
<div class="span12">
<div class="body-content">
<div class="page-header">
- <h1>Building Phoenix Project Web Site</h1>
+ <h1>Prerequisites</h1>
</div>
+<p>The Website build script has severely bit-rotted see <a class="externalLink" href="https://issues.apache.org/jira/browse/PHOENIX-6867">PHOENIX-6867</a></p>
+<ul>
+ <li>The maven site.xml must have proxy set for the <tt>http://conjars.org</tt> org repo</li>
+ <li>The mvn command must be configured to use Java 1.8. Newer versions wonât work.</li>
+</ul>
+<h1>Building Phoenix Project Web Site</h1>
<ol style="list-style-type: decimal">
<li>Make a local copy of source markdown files and html web pages</li>
</ol>
Modified: phoenix/site/publish/classpath_and_url.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/classpath_and_url.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/classpath_and_url.html (original)
+++ phoenix/site/publish/classpath_and_url.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -263,7 +263,7 @@
<div class="section">
<h2 id="Notes">Notes</h2>
<p>Support for <i>master</i> and <i>RPC</i> registries is only available in Phoenix 5.1.4 and later and 5.2.0 and later. Earlier version only support the jdbc:phoenix: protocol variant implementing the the original HBase Zookeeper Connection Registry.</p>
- <p>Support for the registry variants is only available for the HBase versions that support them. Phoenix will throw an error if a variant that the HBase version doesnât support is speci.</p>
+ <p>Support for the registry variants is only available for the HBase versions that support them. Phoenix will throw an error if a variant that the HBase client version doesnât support is specified.</p>
<p>Phoenix 5.2 also supports Hight Availability connections. Documentation for that is only available in the <a class="externalLink" href="https://issues.apache.org/jira/browse/PHOENIX-6491">JIRA ticket</a></p>
</div>
</div>
Modified: phoenix/site/publish/hive_storage_handler.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/hive_storage_handler.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/hive_storage_handler.html (original)
+++ phoenix/site/publish/hive_storage_handler.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -173,54 +173,61 @@
<p>The Apache Phoenix Storage Handler is a plugin that enables Apache Hive access to Phoenix tables from the Apache Hive command line using HiveQL.</p>
<div class="section">
<h2 id="Prerequisites">Prerequisites</h2>
+ <p>This document describes the plugin available in the phoenix-connectors source repo, as of December 2023.</p>
<ul>
- <li>Phoenix 4.8.0+</li>
- <li>Hive 1.2.1+</li>
+ <li>Phoenix 5.1.0+</li>
+ <li>Hive 3.1.2+</li>
+ <li>phoenix-connectors 6.0.0-SNAPSHOT</li>
</ul>
+ <p>The Phoenix Storage handler currently only supports Hive 3.1. It has only been tested with Hive Hive 3.1.3, and Phoenix 5.1.3.</p>
+ <p>A variant for Hive 4 is planned to be provided after Hive 4.0.0 has been released</p>
</div>
<div class="section">
- <h2 id="Hive_Setup">Hive Setup</h2>
- <p>Make phoenix-version-hive.jar available for Hive:</p>
- <p>Step 1: Add to hive-env.sh:</p>
+ <h2 id="Building">Building</h2>
+ <p>The Phoenix Storage Handler used to be part of the main Phoenix repo, but it has been refactored into the separate phoenix-connectors repo after the release of Phoenix 5.0. At the time of writing there is no released version of the connectors project, it must be built from the git source repository HEAD.</p>
+ <p>Official releases will be available at the <a href="download.html">Download</a> page.</p>
+ <p>Check the value of the hbase.version property in the root pom.xml If it is older than HBase 2.5.0, then you need to rebuild HBase locally as described in the BUILDING.md file in the main phoenix repository.</p>
+ <p>Check out the HEAD version of the <a class="externalLink" href="https://github.com/apache/phoenix-connectors">https://github.com/apache/phoenix-connectors</a> repo. Build it with the <tt> mvn clean package </tt> command. The binary distribution will be created in phoenix5-connectors-assembly/target direcory.</p>
+ <p>The driver is built for Hbase 2.4.x by default.</p>
+ <p>To build it for other Hbase versions the <i>hbase.version</i>, <i>hbase.compat.version</i>, <i>hadoop.version</i>, <i>zookeeper.version</i> and <i>hbase-thirdparty-version</i> properties must be to the versions used by HBase. hbase.compat.version is the matching hbase-compat-module in Phoenix, the other versions can be copied from the root pom.xml from the sources of the HBase release.</p>
+ <p>Fore example, to build with HBase 2.1.10 (re-built with -Dhadoop.profile=3.0), use the following command:</p>
<div class="source">
- <pre>HIVE_AUX_JARS_PATH=<path to jar>
+ <pre>mvn clean package -Dhbase.version=2.1.10 -Dhbase.compat.version=2.1.6 -Dhadoop.version=3.0.3 -Dzookeeper.version=3.4.10 -Dhbase-thirdparty-version=2.1.0
</pre>
</div>
- <p>Step 2: Add a property to hive-site.xml so that Hive MapReduce jobs can use the .jar:</p>
+</div>
+<div class="section">
+ <h2 id="Preparing_Hive_3">Preparing Hive 3</h2>
+ <p>Hive 3.1 ships with Hbase 2.0 beta, which is incompatible with Phoenix. To use the Phoenix Storage handler HBase and its dependencies have to be removed from Hive.</p>
+ <p>To remove the shipped HBase 2.0 beta perform the following on each Hive node:</p>
+ <ol style="list-style-type: decimal">
+ <li> <p>Create a backup of the Hive /lib directory</p></li>
+ <li> <p>Remove the HBase dependencies from the /lib directory:</p></li>
+ </ol>
<div class="source">
- <pre><property>
- <name>hive.aux.jars.path</name>
- <value>file://<path></value>
-</property>
+ <pre>mkdir ../lib-removed
+mv hbase-* javax.inject* jcodings-* jersey-* joni-* osgi-resource-locator-* ../lib-removed/
</pre>
</div>
+ <p>Even though Hive ships with HBase jars, it includes a mechanism to load the <tt>hbase mapredcp</tt> jars automatically.</p>
+ <p>To ensure that Hive finds and uses the correct HBase JARs and hbase-site.xml, set the following environment variables in hive-env.sh on each node, or make sure that they are set propery in the enviroment:</p>
+ <p>HBASE_HOME: The root directory of the HBase installation. HBASE_CONF_DIR: The directory where hbase-site.xml resides. Defaults to /etc/hbase/conf, or if it doesnât exist, to $HBASE_HOME/conf. HBASE_BIN_DIR: The directory which holds the <tt>hbase</tt> command. Defaults to $HBASE_HOME/bin.</p>
+ <p>It is assumed that the HADOOP variables are already set correctly, and the HBase libraries and up-date hbase-site.xml are available on each Hive node.</p>
</div>
<div class="section">
- <h2 id="Table_Creation_and_Deletion">Table Creation and Deletion</h2>
- <p>The Phoenix Storage Handler supports both INTERNAL and EXTERNAL Hive tables.</p>
- <div class="section">
- <h3 id="Create_INTERNAL_Table">Create INTERNAL Table</h3>
- <p>For INTERNAL tables, Hive manages the lifecycle of the table and data. When a Hive table is created, a corresponding Phoenix table is also created. Once the Hive table is dropped, the Phoenix table is also deleted.</p>
- <div class="source">
- <pre> create table phoenix_table (
- s1 string,
- i1 int,
- f1 float,
- d1 double
- )
- STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
- TBLPROPERTIES (
- "phoenix.table.name" = "phoenix_table",
- "phoenix.zookeeper.quorum" = "localhost",
- "phoenix.zookeeper.znode.parent" = "/hbase",
- "phoenix.zookeeper.client.port" = "2181",
- "phoenix.rowkeys" = "s1, i1",
- "phoenix.column.mapping" = "s1:s1, i1:i1, f1:f1, d1:d1",
- "phoenix.table.options" = "SALT_BUCKETS=10, DATA_BLOCK_ENCODING='DIFF'"
- );
+ <h2 id="Hive_Setup">Hive Setup</h2>
+ <p>It is necessary to make the <tt>phoenix5-hive-shaded-6.0.0-SNAPSHOT-shaded.jar</tt> available for every hive component. There are many way to achiveve this, one of the simpler ones is to use the <tt>HIVE_AUX_JARS_PATH</tt> environment variable.</p>
+ <p>If hive-env.sh already sets HIVE_AUX_JARS_PATH, then copy the connector JAR there. Otherwise, create a world-readble directory on the system, and copy the connector JAR there. then add a</p>
+ <div class="source">
+ <pre>HIVE_AUX_JARS_PATH=<PATH TO DIRECTORY>
</pre>
- </div>
</div>
+ <p>line to hive-env.sh</p>
+ <p>This must be performed on every Hive node.</p>
+</div>
+<div class="section">
+ <h2 id="Table_Creation_and_Deletion">Table Creation and Deletion</h2>
+ <p>The Phoenix Storage Handler supports only EXTERNAL Hive tables.</p>
<div class="section">
<h3 id="Create_EXTERNAL_Table">Create EXTERNAL Table</h3>
<p>For EXTERNAL tables, Hive works with an existing Phoenix table and manages only Hive metadata. Dropping an EXTERNAL table from Hive deletes only Hive metadata but does not delete the Phoenix table.</p>
@@ -242,6 +249,7 @@ TBLPROPERTIES (
);
</pre>
</div>
+ <p>Specifying the </p>
</div>
<div class="section">
<h3 id="Properties">Properties</h3>
@@ -276,6 +284,7 @@ TBLPROPERTIES (
<li>Mappings between column names for Hive and Phoenix. See <a href="#Limitations">Limitations</a> for details.</li>
</ul></li>
</ol>
+ <p>The phoenix.zookeper properties are optional, if not specified then the the values from hbase-site.xml will be used.</p>
</div>
</div>
<div class="section">
@@ -326,6 +335,7 @@ TBLPROPERTIES (
</tr>
</tbody>
</table>
+ <p>Disabling WAL can lead to data loss.</p>
</div>
<div class="section">
<h3 id="Query_Data">Query Data</h3>
Modified: phoenix/site/publish/language/datatypes.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/language/datatypes.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/datatypes.html (original)
+++ phoenix/site/publish/language/datatypes.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/publish/language/functions.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/language/functions.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/functions.html (original)
+++ phoenix/site/publish/language/functions.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/publish/language/index.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/language/index.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/index.html (original)
+++ phoenix/site/publish/language/index.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/source/src/site/markdown/building_website.md
URL: http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/building_website.md?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/building_website.md (original)
+++ phoenix/site/source/src/site/markdown/building_website.md Fri Dec 1 17:02:03 2023
@@ -1,3 +1,10 @@
+# Prerequisites
+
+The Website build script has severely bit-rotted see [PHOENIX-6867](https://issues.apache.org/jira/browse/PHOENIX-6867)
+
+ - The maven site.xml must have proxy set for the `http://conjars.org` org repo
+ - The mvn command must be configured to use Java 1.8. Newer versions won't work.
+
# Building Phoenix Project Web Site
1. Make a local copy of source markdown files and html web pages
Modified: phoenix/site/source/src/site/markdown/hive_storage_handler.md
URL: http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/hive_storage_handler.md?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/hive_storage_handler.md (original)
+++ phoenix/site/source/src/site/markdown/hive_storage_handler.md Fri Dec 1 17:02:03 2023
@@ -2,55 +2,96 @@
The Apache Phoenix Storage Handler is a plugin that enables Apache Hive access to Phoenix tables from the Apache Hive command line using HiveQL.
+
## Prerequisites
-* Phoenix 4.8.0+
-* Hive 1.2.1+
+This document describes the plugin available in the phoenix-connectors source repo, as of December 2023.
+
+* Phoenix 5.1.0+
+* Hive 3.1.2+
+* phoenix-connectors 6.0.0-SNAPSHOT
+
+The Phoenix Storage handler currently only supports Hive 3.1.
+It has only been tested with Hive Hive 3.1.3, and Phoenix 5.1.3.
+
+A variant for Hive 4 is planned to be provided after Hive 4.0.0 has been released
-## Hive Setup
-Make phoenix-version-hive.jar available for Hive:
+## Building
-Step 1: Add to hive-env.sh:
+The Phoenix Storage Handler used to be part of the main Phoenix repo, but it has been refactored into the separate phoenix-connectors repo after the release of Phoenix 5.0.
+At the time of writing there is no released version of the connectors project, it must be built from the git source repository HEAD.
+Official releases will be available at the [Download](download.html) page.
+
+Check the value of the hbase.version property in the root pom.xml
+If it is older than HBase 2.5.0, then you need to rebuild HBase locally as described
+in the BUILDING.md file in the main phoenix repository.
+
+Check out the HEAD version of the https://github.com/apache/phoenix-connectors repo.
+Build it with the
```
-HIVE_AUX_JARS_PATH=<path to jar>
+mvn clean package
```
+command.
+The binary distribution will be created in phoenix5-connectors-assembly/target direcory.
+
+The driver is built for Hbase 2.4.x by default.
+
+To build it for other Hbase versions the *hbase.version*, *hbase.compat.version*, *hadoop.version*, *zookeeper.version* and *hbase-thirdparty-version* properties must be to the versions used by HBase. hbase.compat.version is the matching hbase-compat-module in Phoenix, the other versions can be copied from the root pom.xml from the sources of the HBase release.
-Step 2: Add a property to hive-site.xml so that Hive MapReduce jobs can use the .jar:
+Fore example, to build with HBase 2.1.10 (re-built with -Dhadoop.profile=3.0), use the following command:
```
-<property>
- <name>hive.aux.jars.path</name>
- <value>file://<path></value>
-</property>
+mvn clean package -Dhbase.version=2.1.10 -Dhbase.compat.version=2.1.6 -Dhadoop.version=3.0.3 -Dzookeeper.version=3.4.10 -Dhbase-thirdparty-version=2.1.0
```
-## Table Creation and Deletion
-The Phoenix Storage Handler supports both INTERNAL and EXTERNAL Hive tables.
-### Create INTERNAL Table
-For INTERNAL tables, Hive manages the lifecycle of the table and data. When a Hive table is created, a corresponding Phoenix table is also created.
-Once the Hive table is dropped, the Phoenix table is also deleted.
+## Preparing Hive 3
-```sql
- create table phoenix_table (
- s1 string,
- i1 int,
- f1 float,
- d1 double
- )
- STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
- TBLPROPERTIES (
- "phoenix.table.name" = "phoenix_table",
- "phoenix.zookeeper.quorum" = "localhost",
- "phoenix.zookeeper.znode.parent" = "/hbase",
- "phoenix.zookeeper.client.port" = "2181",
- "phoenix.rowkeys" = "s1, i1",
- "phoenix.column.mapping" = "s1:s1, i1:i1, f1:f1, d1:d1",
- "phoenix.table.options" = "SALT_BUCKETS=10, DATA_BLOCK_ENCODING='DIFF'"
- );
+Hive 3.1 ships with Hbase 2.0 beta, which is incompatible with Phoenix.
+To use the Phoenix Storage handler HBase and its dependencies have to be removed from Hive.
+
+To remove the shipped HBase 2.0 beta perform the following on each Hive node:
+
+1. Create a backup of the Hive /lib directory
+
+2. Remove the HBase dependencies from the /lib directory:
+
+```
+mkdir ../lib-removed
+mv hbase-* javax.inject* jcodings-* jersey-* joni-* osgi-resource-locator-* ../lib-removed/
+```
+
+Even though Hive ships with HBase jars, it includes a mechanism to load the `hbase mapredcp` jars automatically.
+
+To ensure that Hive finds and uses the correct HBase JARs and hbase-site.xml, set the following
+environment variables in hive-env.sh on each node, or make sure that they are set propery in the enviroment:
+
+HBASE_HOME: The root directory of the HBase installation.
+HBASE_CONF_DIR: The directory where hbase-site.xml resides. Defaults to /etc/hbase/conf, or if it doesn't exist, to $HBASE_HOME/conf.
+HBASE_BIN_DIR: The directory which holds the `hbase` command. Defaults to $HBASE_HOME/bin.
+
+It is assumed that the HADOOP variables are already set correctly, and the HBase libraries and up-date hbase-site.xml are available on each Hive node.
+
+
+## Hive Setup
+
+It is necessary to make the `phoenix5-hive-shaded-6.0.0-SNAPSHOT-shaded.jar` available for every hive component.
+There are many way to achiveve this, one of the simpler ones is to use the `HIVE_AUX_JARS_PATH` environment variable.
+
+If hive-env.sh already sets HIVE_AUX_JARS_PATH, then copy the connector JAR there.
+Otherwise, create a world-readble directory on the system, and copy the connector JAR there. then add a
+
+```
+HIVE_AUX_JARS_PATH=<PATH TO DIRECTORY>
```
+line to hive-env.sh
+
+This must be performed on every Hive node.
+
+## Table Creation and Deletion
+The Phoenix Storage Handler supports only EXTERNAL Hive tables.
### Create EXTERNAL Table
For EXTERNAL tables, Hive works with an existing Phoenix table and manages only Hive metadata. Dropping an EXTERNAL table from Hive deletes only Hive metadata but does not delete the Phoenix table.
@@ -73,6 +114,7 @@ TBLPROPERTIES (
);
```
+Specifying the
### Properties
1. phoenix.table.name
@@ -93,7 +135,7 @@ TBLPROPERTIES (
6. phoenix.column.mapping
* Mappings between column names for Hive and Phoenix. See [Limitations](#Limitations) for details.
-
+The phoenix.zookeper properties are optional, if not specified then the the values from hbase-site.xml will be used.
## Data Ingestion, Deletions, and Updates
Data ingestion can be done by all ways that Hive and Phoenix support:
@@ -126,6 +168,8 @@ phoenix.upsert.batch.size | 1000 | Batch
[phoenix-table-name].disable.wal | false | Temporarily sets the table attribute `DISABLE_WAL` to `true`. Sometimes used to improve performance
[phoenix-table-name].auto.flush | false | When WAL is disabled and if this value is `true`, then MemStore is flushed to an HFile.
+Disabling WAL can lead to data loss.
+
### Query Data
You can use HiveQL for querying data in a Phoenix table. A Hive query on a single table can be as fast as running the query in the Phoenix CLI with the following property settings: `hive.fetch.task.conversion=more` and `hive.exec.parallel=true`
@@ -145,3 +189,4 @@ split.by.stats | false | If true, mapper
## Resources <a id="Resources"></a>
* [PHOENIX-2743] (https://issues.apache.org/jira/browse/PHOENIX-2743) : Implementation, accepted by Apache Phoenix community. Original pull request contains modification for Hive classes.
* [PHOENIX-331] (https://issues.apache.org/jira/browse/PHOENIX-331) : An outdated implementation with support of Hive 0.98.
+