You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by bl...@apache.org on 2019/12/15 19:53:06 UTC
[flink] branch release-1.10 updated: [hotfix][doc][hive] add doc on
what hive dependencies to put in /lib dir
This is an automated email from the ASF dual-hosted git repository.
bli pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new 1cfb233 [hotfix][doc][hive] add doc on what hive dependencies to put in /lib dir
1cfb233 is described below
commit 1cfb2339b0d1c72f8768c3503eff7814b65946d0
Author: bowen.li <bo...@gmail.com>
AuthorDate: Sat Dec 14 22:49:44 2019 -0800
[hotfix][doc][hive] add doc on what hive dependencies to put in /lib dir
---
docs/dev/table/hive/index.md | 79 +++++++++++++++++++++++++++++++++++---
docs/dev/table/hive/index.zh.md | 85 ++++++++++++++++++++++++++++++++++++++---
2 files changed, 153 insertions(+), 11 deletions(-)
diff --git a/docs/dev/table/hive/index.md b/docs/dev/table/hive/index.md
index ede376c..7425467 100644
--- a/docs/dev/table/hive/index.md
+++ b/docs/dev/table/hive/index.md
@@ -29,12 +29,17 @@ under the License.
It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered, defined, and evolved.
Flink offers a two-fold integration with Hive.
-The first is to leverage Hive's Metastore as a persistent catalog for storing Flink specific metadata across sessions.
+
+The first is to leverage Hive's Metastore as a persistent catalog with Flink's `HiveCatalog` for storing Flink specific metadata across sessions.
+For example, users can store their Kafka or ElasticSearch tables in Hive Metastore by using `HiveCatalog`, and reuse them later on in SQL queries.
+
The second is to offer Flink as an alternative engine for reading and writing Hive tables.
-The hive catalog is designed to be “out of the box” compatible with existing Hive installations.
+The `HiveCatalog` is designed to be “out of the box” compatible with existing Hive installations.
You do not need to modify your existing Hive Metastore or change the data placement or partitioning of your tables.
+* Note that we highly recommend users using the [blink planner]({{ site.baseurl }}/dev/table/#dependency-structure) with Hive integration.
+
* This will be replaced by the TOC
{:toc}
@@ -75,7 +80,71 @@ Flink supports the following Hive versions.
### Dependencies
-To integrate with Hive, users need some dependencies in their project. We are using Hive 2.3.4 and 1.2.1 as examples here.
+To integrate with Hive, users need some dependencies in your `/lib/` directory in Flink distribution
+to make the integration work in Table API program or SQL in SQL Client.
+
+We are using Hive 2.3.4 and 1.2.1 as examples here.
+
+
+<div class="codetabs" markdown="1">
+<div data-lang="Hive 2.3.4" markdown="1">
+{% highlight txt %}
+
+/flink-{{ site.version }}
+ /lib
+ flink-dist{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-table{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ // we highly recommend using Flink's blink planner with Hive integration
+ flink-table-blink{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Flink's Hive connector
+ flink-connector-hive{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Hadoop dependencies
+ // Pick the correct Hadoop dependency for your project.
+ // Hive 2.3.4 is built with Hadoop 2.7.2. We pick 2.7.5 which flink-shaded-hadoop is pre-built with,
+ // but users can pick their own hadoop version, as long as it's compatible with Hadoop 2.7.2
+ flink-hadoop-compatibility{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-shaded-hadoop-2-uber-2.7.5-{{ site.shaded_version }}.jar
+
+ // Hive dependencies
+ hive-exec-2.3.4.jar
+
+ ...
+{% endhighlight %}
+</div>
+
+<div data-lang="Hive 1.2.1" markdown="1">
+{% highlight txt %}
+/flink-{{ site.version }}
+ /lib
+ flink-dist{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-table{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ // we highly recommend using Flink's blink planner with Hive integration
+ flink-table-blink{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Flink's Hive connector
+ flink-connector-hive{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Hadoop dependencies
+ // Pick the correct Hadoop dependency for your project.
+ // Hive 1.2.1 is built with Hadoop 2.6.0. We pick 2.6.5 which flink-shaded-hadoop is pre-built with,
+ // but users can pick their own hadoop version, as long as it's compatible with Hadoop 2.6.0
+ flink-hadoop-compatibility{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-shaded-hadoop-2-uber-2.6.5-{{ site.shaded_version }}.jar
+
+ // Hive dependencies
+ hive-metastore-1.2.1.jar
+ hive-exec-1.2.1.jar
+ libfb303-0.9.3.jar
+
+ ...
+{% endhighlight %}
+</div>
+</div>
+
+
+Similarly, If you are building your own program, you need the above dependencies in your mvn file.
<div class="codetabs" markdown="1">
<div data-lang="Hive 2.3.4" markdown="1">
@@ -107,7 +176,7 @@ Hive 2.3.4 is built with Hadoop 2.7.2. We pick 2.7.5 which flink-shaded-hadoop i
<scope>provided</scope>
</dependency>
-<!-- Hive Metastore -->
+<!-- Hive Dependency -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
@@ -145,7 +214,7 @@ but users can pick their own hadoop version, as long as it's compatible with Had
<scope>provided</scope>
</dependency>
-<!-- Hive Metastore -->
+<!-- Hive Dependency -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
diff --git a/docs/dev/table/hive/index.zh.md b/docs/dev/table/hive/index.zh.md
index 4c69f67..7425467 100644
--- a/docs/dev/table/hive/index.zh.md
+++ b/docs/dev/table/hive/index.zh.md
@@ -2,7 +2,7 @@
title: "Hive Integration"
nav-id: hive_tableapi
nav-parent_id: tableapi
-nav-pos: 100
+nav-pos: 110
is_beta: true
nav-show_overview: true
---
@@ -29,12 +29,17 @@ under the License.
It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered, defined, and evolved.
Flink offers a two-fold integration with Hive.
-The first is to leverage Hive's Metastore as a persistent catalog for storing Flink specific metadata across sessions.
+
+The first is to leverage Hive's Metastore as a persistent catalog with Flink's `HiveCatalog` for storing Flink specific metadata across sessions.
+For example, users can store their Kafka or ElasticSearch tables in Hive Metastore by using `HiveCatalog`, and reuse them later on in SQL queries.
+
The second is to offer Flink as an alternative engine for reading and writing Hive tables.
-The hive catalog is designed to be “out of the box” compatible with existing Hive installations.
+The `HiveCatalog` is designed to be “out of the box” compatible with existing Hive installations.
You do not need to modify your existing Hive Metastore or change the data placement or partitioning of your tables.
+* Note that we highly recommend users using the [blink planner]({{ site.baseurl }}/dev/table/#dependency-structure) with Hive integration.
+
* This will be replaced by the TOC
{:toc}
@@ -75,7 +80,71 @@ Flink supports the following Hive versions.
### Dependencies
-To integrate with Hive, users need some dependencies in their project. We are using Hive 2.3.4 and 1.2.1 as examples here.
+To integrate with Hive, users need some dependencies in your `/lib/` directory in Flink distribution
+to make the integration work in Table API program or SQL in SQL Client.
+
+We are using Hive 2.3.4 and 1.2.1 as examples here.
+
+
+<div class="codetabs" markdown="1">
+<div data-lang="Hive 2.3.4" markdown="1">
+{% highlight txt %}
+
+/flink-{{ site.version }}
+ /lib
+ flink-dist{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-table{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ // we highly recommend using Flink's blink planner with Hive integration
+ flink-table-blink{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Flink's Hive connector
+ flink-connector-hive{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Hadoop dependencies
+ // Pick the correct Hadoop dependency for your project.
+ // Hive 2.3.4 is built with Hadoop 2.7.2. We pick 2.7.5 which flink-shaded-hadoop is pre-built with,
+ // but users can pick their own hadoop version, as long as it's compatible with Hadoop 2.7.2
+ flink-hadoop-compatibility{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-shaded-hadoop-2-uber-2.7.5-{{ site.shaded_version }}.jar
+
+ // Hive dependencies
+ hive-exec-2.3.4.jar
+
+ ...
+{% endhighlight %}
+</div>
+
+<div data-lang="Hive 1.2.1" markdown="1">
+{% highlight txt %}
+/flink-{{ site.version }}
+ /lib
+ flink-dist{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-table{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ // we highly recommend using Flink's blink planner with Hive integration
+ flink-table-blink{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Flink's Hive connector
+ flink-connector-hive{{ site.scala_version_suffix }}-{{ site.version }}.jar
+
+ // Hadoop dependencies
+ // Pick the correct Hadoop dependency for your project.
+ // Hive 1.2.1 is built with Hadoop 2.6.0. We pick 2.6.5 which flink-shaded-hadoop is pre-built with,
+ // but users can pick their own hadoop version, as long as it's compatible with Hadoop 2.6.0
+ flink-hadoop-compatibility{{ site.scala_version_suffix }}-{{ site.version }}.jar
+ flink-shaded-hadoop-2-uber-2.6.5-{{ site.shaded_version }}.jar
+
+ // Hive dependencies
+ hive-metastore-1.2.1.jar
+ hive-exec-1.2.1.jar
+ libfb303-0.9.3.jar
+
+ ...
+{% endhighlight %}
+</div>
+</div>
+
+
+Similarly, If you are building your own program, you need the above dependencies in your mvn file.
<div class="codetabs" markdown="1">
<div data-lang="Hive 2.3.4" markdown="1">
@@ -107,7 +176,7 @@ Hive 2.3.4 is built with Hadoop 2.7.2. We pick 2.7.5 which flink-shaded-hadoop i
<scope>provided</scope>
</dependency>
-<!-- Hive Metastore -->
+<!-- Hive Dependency -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
@@ -145,7 +214,7 @@ but users can pick their own hadoop version, as long as it's compatible with Had
<scope>provided</scope>
</dependency>
-<!-- Hive Metastore -->
+<!-- Hive Dependency -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
@@ -272,6 +341,10 @@ Currently `HiveCatalog` supports most Flink data types with the following mappin
<td class="text-center">DATE</td>
</tr>
<tr>
+ <td class="text-center">TIMESTAMP</td>
+ <td class="text-center">TIMESTAMP</td>
+ </tr>
+ <tr>
<td class="text-center">BYTES</td>
<td class="text-center">BINARY</td>
</tr>