You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by gi...@apache.org on 2026/09/11 18:26:56 UTC
(iceberg-python) branch gh-readonly-queue/main/pr-3927-5109df4cf11db5d00d8d9e7c613d8389342cb4a1 created (now 41a40312d)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-3927-5109df4cf11db5d00d8d9e7c613d8389342cb4a1
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
at 41a40312d Hive: document hive2-compatible and fix commit locks on Hive 2.1 (#3927)
This branch includes the following new commits:
new 41a40312d Hive: document hive2-compatible and fix commit locks on Hive 2.1 (#3927)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(iceberg-python) 01/01: Hive: document hive2-compatible and fix commit locks on Hive 2.1 (#3927)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-3927-5109df4cf11db5d00d8d9e7c613d8389342cb4a1
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 41a40312d0f35e6e36d5cb6dd44f2944bd715856
Author: Kevin Liu <ke...@users.noreply.github.com>
AuthorDate: Fri Sep 11 18:26:32 2026 +0000
Hive: document hive2-compatible and fix commit locks on Hive 2.1 (#3927)
* Docs: Describe hive.hive2-compatible as the Hive 2 setting
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
* Hive: Mark the commit lock as non-transactional
Hive 2.1.0 rejects a lock component whose operation type arrives as the
default UNSET, and the generated Thrift client always sends that
default. The commit lock is not part of a metastore transaction, so
send NO_TXN, which every Hive release accepts.
Verified against Hive 2.0.0, 2.1.0, 2.3.2, 3.1.3, 4.0.0, 4.0.1, 4.1.0
and 4.2.1 metastores.
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
* Hive: Name the Hive versions that lack get_table_req
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
* Clarify comments on lock request operation type
Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
* Wrap lock request comment to fit line length
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
* thanks geruh
---------
Co-authored-by: Claude Fable 5.1 <no...@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
---
mkdocs/docs/configuration.md | 4 ++--
pyiceberg/catalog/hive.py | 15 +++++++++++++--
2 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md
index fd11762d1..18f9db973 100644
--- a/mkdocs/docs/configuration.md
+++ b/mkdocs/docs/configuration.md
@@ -775,12 +775,12 @@ catalog:
| Key | Example | Description |
|------------------------------| ------- | ------------------------------------ |
-| hive.hive2-compatible | true | Using Hive 2.x compatibility mode |
+| hive.hive2-compatible | true | Set to `true` when using a Hive 2.x metastore |
| hive.kerberos-authentication | true | Using authentication via Kerberos |
| hive.kerberos-service-name | hive | Kerberos service name (default hive) |
| ugi | t-1234:secret | Hadoop UGI for Hive client. |
-When using Hive 2.x, make sure to set the compatibility flag:
+Hive 3 and newer need no extra configuration. When using a Hive 2.x metastore, set the compatibility flag:
```yaml
catalog:
diff --git a/pyiceberg/catalog/hive.py b/pyiceberg/catalog/hive.py
index 94213c833..ed6d3c6fd 100644
--- a/pyiceberg/catalog/hive.py
+++ b/pyiceberg/catalog/hive.py
@@ -30,6 +30,7 @@ from hive_metastore.ThriftHiveMetastore import Client
from hive_metastore.ttypes import (
AlreadyExistsException,
CheckLockRequest,
+ DataOperationType,
EnvironmentContext,
FieldSchema,
GetTableRequest,
@@ -383,7 +384,7 @@ class HiveCatalog(MetastoreCatalog):
raise TableAlreadyExistsError(f"Table {hive_table.dbName}.{hive_table.tableName} already exists") from e
def _fetch_hive_table(self, open_client: Client, database_name: str, table_name: str) -> HiveTable:
- # Hive 4.0.1 removed get_table, and Hive 2 does not have get_table_req
+ # Hive 4.0.1 removed get_table, and Hive 2.2 and older do not have get_table_req
if self._hive2_compatible:
return open_client.get_table(dbname=database_name, tbl_name=table_name)
return open_client.get_table_req(GetTableRequest(dbName=database_name, tblName=table_name)).table
@@ -506,8 +507,18 @@ class HiveCatalog(MetastoreCatalog):
raise NotImplementedError
def _create_lock_request(self, database_name: str, table_name: str) -> LockRequest:
+ # Iceberg commits are not executed within a Hive transaction, so the lock component uses operationType=NO_TXN.
+ # Setting it explicitly also matters for Hive 2.1.0, which rejects a lock component left at the default UNSET
+ # operation type. Hive 2.1.1 relaxed this validation:
+ # https://github.com/apache/hive/blob/rel/release-2.1.1/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L939-L947
+ # operation type.
lock_component: LockComponent = LockComponent(
- level=LockLevel.TABLE, type=LockType.EXCLUSIVE, dbname=database_name, tablename=table_name, isTransactional=True
+ level=LockLevel.TABLE,
+ type=LockType.EXCLUSIVE,
+ dbname=database_name,
+ tablename=table_name,
+ operationType=DataOperationType.NO_TXN,
+ isTransactional=True,
)
lock_request: LockRequest = LockRequest(component=[lock_component], user=getpass.getuser(), hostname=socket.gethostname())