You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/09/03 06:47:23 UTC
(kyuubi) branch master updated: [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 8cafd873c1 [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
8cafd873c1 is described below
commit 8cafd873c11323310d306481368290780a37d5b7
Author: maomaodev <li...@tencent.com>
AuthorDate: Thu Sep 3 14:47:14 2026 +0800
[KYUUBI #7664][DOC] Reformat and improve docs for KSHC
### Why are the changes needed?
1. Reformat KSHC docs from RST to Markdown, as described in the issue https://github.com/apache/kyuubi/issues/7434.
2. Enrich the description for KSCH.
### How was this patch tested?
Tested by building the documentation and comparing the pages against the live site:
```
open https://kyuubi.readthedocs.io/en/master/connector/spark/hive.html
open _build/html/connector/spark/hive.html
```
### Was this patch assisted by generative AI tooling?
Assisted-by: DeepSeek-V4-Pro
Closes #7667 from maomaodev/kyuubi-7664.
Closes #7664
85a387162 [Cheng Pan] Update docs/connector/spark/hive.md
c3edb02cd [Cheng Pan] Update docs/connector/spark/hive.md
d8ded95cb [lifumao] [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
d8074cfc3 [lifumao] [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
b8dc3b759 [lifumao] [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
207436d9a [lifumao] [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
85a67d0c4 [lifumao] [KYUUBI #7664][DOC] Reformat and improve docs for KSHC
Lead-authored-by: maomaodev <li...@tencent.com>
Co-authored-by: Cheng Pan <pa...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
docs/connector/spark/hive.md | 172 ++++++++++++++++++++++++++++++++++++++++++
docs/connector/spark/hive.rst | 133 --------------------------------
2 files changed, 172 insertions(+), 133 deletions(-)
diff --git a/docs/connector/spark/hive.md b/docs/connector/spark/hive.md
new file mode 100644
index 0000000000..5cfd4a41ca
--- /dev/null
+++ b/docs/connector/spark/hive.md
@@ -0,0 +1,172 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Hive
+
+You may know that the Apache Spark has built-in support for accessing Hive tables, it works well in most cases,
+but is limited to one Hive Metastore. The Kyuubi Spark Hive connector (KSHC) implements a Hive connector based
+on Spark DataSource V2 API, supports accessing multiple Hive Metastore in a single Spark application.
+
+## Hive Integration
+
+To enable the integration of Kyuubi Spark SQL engine and Hive connector through
+Spark DataSource V2 API, you need to:
+
+- Referencing the Hive connector [dependencies](#kshc-dependencies)
+- Setting the Spark catalog [configurations](#kshc-configurations)
+
+(kshc-dependencies)=
+
+### Dependencies
+
+The **classpath** of Kyuubi Spark SQL engine with Hive connector supported consists of
+
+1. kyuubi-spark-sql-engine_2.12-{{ release }}.jar, the engine jar deployed with a Kyuubi distribution
+2. a copy of Spark distribution
+3. kyuubi-spark-connector-hive_2.12-{{ release }}.jar, which can be found in the [Maven Central](https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-connector-hive)
+
+In order to make the Hive connector packages visible for the runtime classpath of engines, we can use one of these methods:
+
+1. Put the Kyuubi Hive connector packages into `$SPARK_HOME/jars` directly
+2. Set `spark.jars=/path/to/kyuubi-spark-connector-hive_2.12-<version>.jar`
+
+```{note}
+Starting from v1.9.2 and v1.10.0, KSHC jars available in the [Maven Central](https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-connector-hive) guarantee binary compatibility across
+multiple Spark versions, so a single jar works across them without rebuilding.
+```
+
+(kshc-configurations)=
+
+### Configurations
+
+To activate functionality of Kyuubi Spark Hive connector, we can set the following configurations:
+
+```properties
+spark.sql.catalog.hive_catalog org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
+spark.sql.catalog.hive_catalog.hive.metastore.uris thrift://metastore-host:port
+spark.sql.catalog.hive_catalog.<other.hive.conf> <value>
+spark.sql.catalog.hive_catalog.<other.hadoop.conf> <value>
+```
+
+Besides the catalog-level configurations above, the Kyuubi Spark Hive connector provides the following configurations:
+
+| Key | Default | Meaning | Type | Since |
+|----------------------------------------------------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|--------|
+| `spark.sql.kyuubi.hive.connector.externalCatalog.share.policy` | `ONE_FOR_ALL` | The share policy for the external catalog. `ONE_FOR_ONE` means an external catalog is used by only one `HiveTableCatalog`, while `ONE_FOR_ALL` shares an external catalog globally across the catalogs with the same name. | string | 1.7.0 |
+| `spark.sql.catalog.<catalog>.delegation.token.renewal.enabled` | `true` | Whether to enable delegation token renewal for the Kerberized Hive Metastore of this catalog. | boolean | 1.8.0 |
+| `spark.sql.kyuubi.hive.connector.read.convertMetastoreParquet` | `true` | When enabled, the data source Parquet reader is used to process Parquet tables created by HiveQL syntax, instead of the Hive SerDe. | boolean | 1.11.0 |
+| `spark.sql.kyuubi.hive.connector.read.convertMetastoreOrc` | `true` | When enabled, the data source ORC reader is used to process ORC tables created by HiveQL syntax, instead of the Hive SerDe. | boolean | 1.11.0 |
+| `spark.sql.kyuubi.hive.connector.dropTableAsPurgeTable` | `false` | When enabled, `DROP TABLE` completely removes its data by skipping HDFS trash, equivalent to the `PURGE TABLE` command. | boolean | 1.12.0 |
+| `spark.sql.catalog.<catalog>.hive.metastore.warehouse.dir` | <undefined> | The default warehouse directory for the catalog, taking precedence over the global `spark.sql.warehouse.dir` when creating a database. | string | 1.12.0 |
+
+```{note}
+Catalog-level configurations (`spark.sql.catalog.<catalog>.*`) are captured when the catalog
+is lazily initialized on first access, so they take effect as long as they are set before that.
+Once the catalog is initialized, subsequent `SET` on these keys no longer affects it. Note that
+under the default `ONE_FOR_ALL` share policy, configurations only apply to the first initialized
+Hive client instance shared across catalogs with the same name.
+
+In Spark cluster deploy mode with Kerberos and no keytab configured, all catalogs that
+need HMS access must be declared with their `hive.metastore.uris` at the Spark application
+bootstrap (e.g. in `spark-defaults.conf` or via `--conf`), so that Spark can fetch HMS delegation
+tokens ahead of time during submission and distribute them to executors.
+```
+
+## Hive Connector Operations
+
+Taking `CREATE NAMESPACE` as an example,
+
+```sql
+CREATE NAMESPACE ns;
+```
+
+Taking `CREATE TABLE` as an example,
+
+```sql
+CREATE TABLE hive_catalog.ns.foo (
+ id bigint COMMENT 'unique id',
+ data string)
+USING parquet;
+```
+
+Taking `SELECT` as an example,
+
+```sql
+SELECT * FROM hive_catalog.ns.foo;
+```
+
+Taking `INSERT` as an example,
+
+```sql
+INSERT INTO hive_catalog.ns.foo VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+Taking `DROP TABLE` as an example,
+
+```sql
+DROP TABLE hive_catalog.ns.foo;
+```
+
+Taking `DROP NAMESPACE` as an example,
+
+```sql
+DROP NAMESPACE hive_catalog.ns;
+```
+
+## Features
+
+Since v1.11.0, KSHC uses the data source Parquet/ORC reader to process Parquet and ORC tables
+created by HiveQL syntax by default, instead of the Hive SerDe. This enables vectorized reading.
+Set `spark.sql.kyuubi.hive.connector.read.convertMetastoreParquet` and
+`spark.sql.kyuubi.hive.connector.read.convertMetastoreOrc` to `false` to fall back to the Hive
+SerDe.
+
+Since v1.12.0, KSHC supports the `PURGE TABLE` command. By default, `DROP TABLE` moves the table
+data to the HDFS trash. Set `spark.sql.kyuubi.hive.connector.dropTableAsPurgeTable` to `true` to
+make `DROP TABLE` skip the HDFS trash and completely remove its data, behaving like `PURGE TABLE`.
+
+Since v1.13.0, KSHC supports Dynamic Partition Pruning (DPP) for partitioned Hive tables, which
+significantly reduces the amount of data scanned when joining against large partitioned tables.
+
+## Advanced Usages
+
+Though KSHC is a pure Spark DataSource V2 connector which isn't coupled with Kyuubi deployment, due to the
+implementation inside `spark-sql`, you should not expect KSHC works properly with `spark-sql`, and
+any issues caused by such a combination usage won't be considered at this time. Instead, it's recommended
+using BeeLine with Kyuubi as a drop-in replacement for `spark-sql`, or switching to `spark-shell`.
+
+KSHC supports accessing Kerberized Hive Metastore and HDFS, by using keytab, or TGT cache, or Delegation Token.
+It's not expected to work properly with multiple KDC instances, the limitation comes from JDK Krb5LoginModule,
+for such cases, consider setting up Cross-Realm Kerberos trusts, then you just need to talk with one KDC.
+
+For HMS Thrift API used by Spark, it's known that Hive 2.3.9 client is compatible with HMS from 2.1 to 3.1, and
+Hive 2.3.10 client is compatible with HMS from 1.1 to 3.1, such version combinations should cover the most cases.
+For other corner cases, KSHC also supports `spark.sql.catalog.<catalog_name>.spark.sql.hive.metastore.jars` and
+`spark.sql.catalog.<catalog_name>.spark.sql.hive.metastore.version` as well as the Spark built-in Hive datasource
+does, you can refer to the Spark documentation for details.
+
+## Limitations
+
+Currently, KSHC has the following limitations:
+
+- Persistent Hive views and UDFs are not supported through a KSHC catalog, so `CREATE VIEW`,
+ `SHOW VIEWS`, `CREATE FUNCTION`, and `SHOW FUNCTIONS` all fail when the target or current
+ catalog is a KSHC catalog. Temporary views and temporary functions are not affected. As a
+ workaround, persistent views and UDFs can be created and accessed through the Spark built-in
+ `spark_catalog`.
+- Bucket tables are not supported and are handled as regular Hive tables.
+
diff --git a/docs/connector/spark/hive.rst b/docs/connector/spark/hive.rst
deleted file mode 100644
index 1152e4c472..0000000000
--- a/docs/connector/spark/hive.rst
+++ /dev/null
@@ -1,133 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-`Hive`_
-==========
-
-You may know that the Apache Spark has built-in support for accessing Hive tables, it works well in most cases,
-but is limited to one Hive Metastore. The Kyuubi Spark Hive connector(KSHC) implemented a Hive connector based
-on Spark DataSource V2 API, supports accessing multiple Hive Metastore in a single Spark application.
-
-Hive Integration
-----------------
-
-To enable the integration of Kyuubi Spark SQL engine and Hive connector through
-Spark DataSource V2 API, you need to:
-
-- Referencing the Hive connector :ref:`dependencies<kyuubi-hive-deps>`
-- Setting the Spark catalog :ref:`configurations<kyuubi-hive-conf>`
-
-.. _kyuubi-hive-deps:
-
-Dependencies
-************
-
-The **classpath** of Kyuubi Spark SQL engine with Hive connector supported consists of
-
-1. kyuubi-spark-sql-engine-\ |release|\ _2.12.jar, the engine jar deployed with a Kyuubi distribution
-2. a copy of Spark distribution
-3. kyuubi-spark-connector-hive_2.12-\ |release|\ , which can be found in the `Maven Central`_
-
-In order to make the Hive connector packages visible for the runtime classpath of engines, we can use one of these methods:
-
-1. Put the Kyuubi Hive connector packages into ``$SPARK_HOME/jars`` directly
-2. Set ``spark.jars=/path/to/kyuubi-hive-connector``
-
-.. note::
- Starting from v1.9.2 and v1.10.0, KSHC jars available in the `Maven Central`_ guarantee binary compatibility across
- Spark versions, namely, Spark 3.3 onwards.
-
-.. _kyuubi-hive-conf:
-
-Configurations
-**************
-
-To activate functionality of Kyuubi Spark Hive connector, we can set the following configurations:
-
-.. code-block:: properties
-
- spark.sql.catalog.hive_catalog org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
- spark.sql.catalog.hive_catalog.hive.metastore.uris thrift://metastore-host:port
- spark.sql.catalog.hive_catalog.<other.hive.conf> <value>
- spark.sql.catalog.hive_catalog.<other.hadoop.conf> <value>
-
-Hive Connector Operations
--------------------------
-
-Taking ``CREATE NAMESPACE`` as a example,
-
-.. code-block:: sql
-
- CREATE NAMESPACE ns;
-
-Taking ``CREATE TABLE`` as a example,
-
-.. code-block:: sql
-
- CREATE TABLE hive_catalog.ns.foo (
- id bigint COMMENT 'unique id',
- data string)
- USING parquet;
-
-Taking ``SELECT`` as a example,
-
-.. code-block:: sql
-
- SELECT * FROM hive_catalog.ns.foo;
-
-Taking ``INSERT`` as a example,
-
-.. code-block:: sql
-
- INSERT INTO hive_catalog.ns.foo VALUES (1, 'a'), (2, 'b'), (3, 'c');
-
-Taking ``DROP TABLE`` as a example,
-
-.. code-block:: sql
-
- DROP TABLE hive_catalog.ns.foo;
-
-Taking ``DROP NAMESPACE`` as a example,
-
-.. code-block:: sql
-
- DROP NAMESPACE hive_catalog.ns;
-
-Advanced Usages
-***************
-
-Though KSHC is a pure Spark DataSource V2 connector which isn't coupled with Kyuubi deployment, due to the
-implementation inside ``spark-sql``, you should not expect KSHC works properly with ``spark-sql``, and
-any issues caused by such a combination usage won't be considered at this time. Instead, it's recommended
-using BeeLine with Kyuubi as a drop-in replacement for ``spark-sql``, or switching to ``spark-shell``.
-
-KSHC supports accessing Kerberized Hive Metastore and HDFS, by using keytab, or TGT cache, or Delegation Token.
-It's not expected to work properly with multiple KDC instances, the limitation comes from JDK Krb5LoginModule,
-for such cases, consider setting up Cross-Realm Kerberos trusts, then you just need to talk with one KDC.
-
-For HMS Thrift API used by Spark, it's known that Hive 2.3.9 client is compatible with HMS from 2.1 to 4.0, and
-Hive 2.3.10 client is compatible with HMS from 1.1 to 4.0, such version combinations should cover the most cases.
-For other corner cases, KSHC also supports ``spark.sql.catalog.<catalog_name>.spark.sql.hive.metastore.jars`` and
-``spark.sql.catalog.<catalog_name>.spark.sql.hive.metastore.version`` as well as the Spark built-in Hive datasource
-does, you can refer to the Spark documentation for details.
-
-Currently, KSHC has not implemented the Parquet/ORC Hive tables read/write optimization, in other words, it always
-uses Hive SerDe to access Hive tables, so there might be a performance gap compared to the Spark built-in Hive
-datasource, especially due to lack of support for vectorized reading. And you may hit bugs caused by Hive SerDe,
-e.g. ``ParquetHiveSerDe`` can not read Parquet files that decimals are written in int-based format produced by
-Spark Parquet datasource writer with ``spark.sql.parquet.writeLegacyFormat=false``.
-
-.. _Apache Spark: https://spark.apache.org/
-.. _Maven Central: https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-connector-hive