You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by ad...@apache.org on 2025/01/11 17:27:08 UTC
(ozone) branch master updated: HDDS-11962. [Docs] Hive Integration (#7596)
This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 19c8136afe HDDS-11962. [Docs] Hive Integration (#7596)
19c8136afe is described below
commit 19c8136afe34683193f9cdeef0b369a54c1ad045
Author: Wei-Chiu Chuang <we...@apache.org>
AuthorDate: Sat Jan 11 09:27:03 2025 -0800
HDDS-11962. [Docs] Hive Integration (#7596)
---
hadoop-hdds/docs/content/integration/Hive.md | 169 +++++++++++++++++++++++++
hadoop-hdds/docs/content/integration/_index.md | 26 ++++
2 files changed, 195 insertions(+)
diff --git a/hadoop-hdds/docs/content/integration/Hive.md b/hadoop-hdds/docs/content/integration/Hive.md
new file mode 100644
index 0000000000..8b43236d56
--- /dev/null
+++ b/hadoop-hdds/docs/content/integration/Hive.md
@@ -0,0 +1,169 @@
+---
+title: Hive
+weight: 4
+menu:
+ main:
+ parent: "Application Integrations"
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Apache Hive has supported Apache Ozone since Hive 4.0. To enable Hive to work with Ozone paths, ensure that the `ozone-filesystem-hadoop3` JAR is added to the Hive classpath.
+
+## Supported Access Protocols
+
+Hive supports the following protocols for accessing Ozone data:
+
+* ofs
+* o3fs
+* s3a
+
+## Supported Replication Types
+
+Hive is compatible with Ozone buckets configured with either:
+
+* RATIS (Replication)
+* Erasure Coding
+
+## Accessing Ozone Data in Hive
+
+Hive provides two methods to interact with data in Ozone:
+
+* Managed Tables
+* External Tables
+
+### Managed Tables
+#### Configuring the Hive Warehouse Directory in Ozone
+To store managed tables in Ozone, update the following properties in the `hive-site.xml` configuration file:
+
+```xml
+<property>
+ <name>hive.metastore.warehouse.dir</name>
+ <value>ofs://ozone1/vol1/bucket1/warehouse/</value>
+</property>
+```
+
+#### Creating a Managed Table
+You can create a managed table with a standard `CREATE TABLE` statement:
+
+```sql
+CREATE TABLE myTable (
+ id INT,
+ name STRING
+);
+```
+
+#### Loading Data into a Managed Table
+Data can be loaded into a Hive table from an Ozone location:
+
+```sql
+LOAD DATA INPATH 'ofs://ozone1/vol1/bucket1/table.csv' INTO TABLE myTable;
+```
+
+#### Specifying a Custom Ozone Path
+You can define a custom Ozone path for a database using the `MANAGEDLOCATION` clause:
+
+```sql
+CREATE DATABASE d1 MANAGEDLOCATION 'ofs://ozone1/vol1/bucket1/data';
+```
+
+Tables created in the database d1 will be stored under the specified path:
+`ofs://ozone1/vol1/bucket1/data`
+
+#### Verifying the Ozone Path
+You can confirm that Hive references the correct Ozone path using:
+
+```sql
+SHOW CREATE DATABASE d1;
+```
+
+Output Example:
+
+```text
++----------------------------------------------------+
+| createdb_stmt |
++----------------------------------------------------+
+| CREATE DATABASE `d1` |
+| LOCATION |
+| 'ofs://ozone1/vol1/bucket1/external/d1.db' |
+| MANAGEDLOCATION |
+| 'ofs://ozone1/vol1/bucket1/data' |
++----------------------------------------------------+
+```
+
+### External Tables
+
+Hive allows the creation of external tables to query existing data stored in Ozone.
+
+#### Creating an External Table
+```sql
+CREATE EXTERNAL TABLE external_table (
+ id INT,
+ name STRING
+)
+LOCATION 'ofs://ozone1/vol1/bucket1/table1';
+```
+
+* With external tables, the data is expected to be created and managed by another tool.
+* Hive queries the data as-is.
+* Note: Dropping an external table in Hive does not delete the associated data.
+
+To set a default path for external tables, configure the following property in the `hive-site.xml` file:
+```xml
+<property>
+ <name>hive.metastore.warehouse.external.dir</name>
+ <value>ofs://ozone1/vol1/bucket1/external/</value>
+</property>
+```
+This property specifies the base directory for external tables when no explicit `LOCATION` is provided.
+
+#### Verifying the External Table Path
+To confirm the table's metadata and location, use:
+
+```sql
+SHOW CREATE TABLE external_table;
+```
+Output Example:
+
+```text
++----------------------------------------------------+
+| createtab_stmt |
++----------------------------------------------------+
+| CREATE EXTERNAL TABLE `external_table`( |
+| `id` int, |
+| `name` string) |
+| ROW FORMAT SERDE |
+| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
+| STORED AS INPUTFORMAT |
+| 'org.apache.hadoop.mapred.TextInputFormat' |
+| OUTPUTFORMAT |
+| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
+| LOCATION |
+| 'ofs://ozone1/vol1/bucket1/table1' |
+| TBLPROPERTIES ( |
+| 'bucketing_version'='2', |
+| 'transient_lastDdlTime'='1734725573') |
++----------------------------------------------------+
+```
+
+## Using the S3A Protocol
+In addition to ofs, Hive can access Ozone using the S3 Gateway via the S3A file system.
+
+For more information, consult:
+
+* The [S3 Protocol]({{< ref "interface/S3.md">}})
+* The [Hadoop S3A](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html) documentation.
diff --git a/hadoop-hdds/docs/content/integration/_index.md b/hadoop-hdds/docs/content/integration/_index.md
new file mode 100644
index 0000000000..87f6a4825b
--- /dev/null
+++ b/hadoop-hdds/docs/content/integration/_index.md
@@ -0,0 +1,26 @@
+---
+title: "Application Integrations"
+menu:
+ main:
+ weight: 5
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+{{<jumbotron>}}
+Many applications can be integrated with Ozone through the Hadoop-compatible ofs interface or the S3 interface.
+{{</jumbotron>}}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org