You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/13 06:30:00 UTC
[jira] [Updated] (HIVE-29843) Concurrent create_database requests may recursively delete a shared database directory
[ https://issues.apache.org/jira/browse/HIVE-29843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HIVE-29843:
----------------------------------
Labels: pull-request-available (was: )
> Concurrent create_database requests may recursively delete a shared database directory
> --------------------------------------------------------------------------------------
>
> Key: HIVE-29843
> URL: https://issues.apache.org/jira/browse/HIVE-29843
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Affects Versions: 4.0.0-beta-1
> Environment: Hive Metastore: 4.0.0-beta-1
> MySQL: 8.0.30
> Hadoop: 3.3.1
> FileSystem: S3A-compatible object storage
> Standalone reproduction object storage: MinIO
> Iceberg: 1.6.1
> Concurrency: 16 independent Hive Metastore clients
> Reporter: Zhenqi Li
> Assignee: Zhenqi Li
> Priority: Major
> Labels: pull-request-available
>
> When multiple clients concurrently initialize the same Iceberg namespace, Hive Metastore may process identical {{create_database}} requests and create the same database directory. Only one request can successfully insert the database metadata; the other requests fail because of the database unique constraint.
> The failed requests then use a local {{madeDir}} or {{madeExternalDir}} flag to decide whether to recursively clean up the entire database directory. However, this flag only records that the request created or observed a usable directory; it does not prove that the request still has exclusive ownership of that directory. Another request may already have committed metadata for the same path or written data below it.
> As a result, a failed {{create_database}} request may move or delete a shared database directory, leaving valid Metastore metadata pointing to missing object-storage files.
> h2. Affected Versions
> The issue was reproduced against Hive Metastore {{{}4.0.0-beta-1{}}}.
> Source inspection found the same directory-ownership and rollback-cleanup pattern in Hive branches from {{1.0}} through {{{}4.2.1{}}}. I did not find an upstream fix specifically addressing the {{create_database}} directory ownership race.
> h2. Description
> Multiple clients initializing the same Iceberg namespace can concurrently call Hive Metastore with identical {{create_database}} requests.
> A typical target path is:
> {code:java}
> s3a://warehouse/iceberg/test1.db
> {code}
> The following sequence can occur:
> # Multiple requests check that the database does not exist.
> # Multiple requests check that the target directory does not exist.
> # Multiple requests call {{mkdirs}} for the same path.
> # One or more requests record {{madeDir}} or {{madeExternalDir}} as {{true}} before the metadata winner is determined.
> # One request successfully inserts the database into the Metastore {{DBS}} table.
> # The other requests fail because of the {{DBS.UNIQUE_DATABASE}} unique constraint.
> # The failed requests enter the rollback cleanup path.
> # Based on their local flag, they assume that they own the database directory.
> # A failed request recursively moves or deletes the entire database directory.
> There is also a more general interleaving, similar to the one documented in HIVE-23437:
> # Request R1 creates the directory and is paused before committing metadata.
> # Request R2 observes that the directory already exists, so R2 does not create it.
> # R2 successfully commits database metadata referencing that directory.
> # R1 resumes, loses the metadata race, and rolls back.
> # R1 still has {{madeDir=true}} and recursively deletes the directory already adopted by R2.
> This second interleaving shows that even a perfectly atomic {{mkdir}} implementation would not be sufficient. Directory creation provenance is not equivalent to current deletion ownership.
> h2. Steps to Reproduce
> h3. 1. Prepare Hive Metastore
> Start Hive Metastore with a MySQL metadata database and S3A-compatible object storage. Confirm that the target database path does not exist.
> For example:
> {code:java}
> s3a://warehouse/hms_direct_race.db
> {code}
> h3. 2. Concurrently invoke create_database
> Use multiple independent Hive Metastore clients and concurrently call:
> {code:java}
> client.createDatabase(database);
> {code}
> All requests must use the same catalog, database name, and location. The standalone reproduction used:
> {code:java}
> catalog: hive
> database: hms_direct_race
> concurrency: 16
> {code}
> h3. 3. Observe the Metastore result
> The reproduced result was:
> * 1 request succeeded.
> * 15 requests failed.
> * The failed requests reported a MySQL unique-key conflict:
> {code:java}
> Duplicate entry 'hms_direct_race-hive'
> for key 'DBS.UNIQUE_DATABASE'
> {code}
> * Multiple handlers logged that the same database path was created.
> * Failed requests entered the database-directory cleanup path.
> h3. 4. Observe directory cleanup
> If another request writes content below the database directory before rollback cleanup runs, the failed request may move or delete that content together with the entire database directory.
> For example, affected Iceberg content may include:
> {code:java}
> iceberg/test1.db/test3/metadata/00000-....metadata.json
> {code}
> When Hadoop Trash is enabled, the path may be moved under a location similar to:
> {code:java}
> s3a://warehouse/user/root/.Trash/Current/iceberg/test1.db
> {code}
> h2. Reproduction Log Excerpts
> The following logs are from a standalone reproduction that bypassed all higher-level application code. It used 16 independent Hive Metastore clients. The run started at {{2026-08-27 02:23:40 UTC}} and targeted database {{{}hms_direct_race_20260827T022340Z{}}}.
> h3. Client result
> {code:java}
> HMS_CREATE_DATABASE_RACE host=hive-metastore port=9083 database=hms_direct_race_20260827T022340Z workers=16 location=s3a://warehouse/hms_direct_race_20260827T022340Z.db
> worker=00 result=FAILURE error=MetaException: JDODataStoreException: Exception thrown flushing changes to datastore Root cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> worker=01 result=FAILURE error=MetaException: JDODataStoreException: Exception thrown flushing changes to datastore Root cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> ...
> worker=11 result=SUCCESS
> ...
> worker=15 result=FAILURE error=MetaException: JDODataStoreException: Exception thrown flushing changes to datastore Root cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> HMS_CREATE_DATABASE_RACE_COMPLETE elapsed_ms=10380
> {code}
> The final result was one successful request and 15 failures caused by {{{}DBS.UNIQUE_DATABASE{}}}.
> h3. HMS audit and directory creation
> Hive Metastore audit logs recorded multiple concurrent {{create_database}} requests for the same database. One example is:
> {code:java}
> 2026-08-27T02:23:48,142 INFO [Metastore-Handler-Pool: Thread-1341] HiveMetaStore.audit: ugi=root ip=172.19.0.6 cmd=source:172.19.0.6 create_database: Database(name:hms_direct_race_20260827T022340Z, description:direct concurrent create_database test, locationUri:s3a://warehouse/hms_direct_race_20260827T022340Z.db, parameters:{}, catalogName:hive)
> {code}
> Multiple handlers then processed the same database path:
> {code:java}
> 2026-08-27T02:23:48,401 INFO [Metastore-Handler-Pool: Thread-1340] HMSHandler: Creating database path in external directory s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,403 INFO [Metastore-Handler-Pool: Thread-1334] HMSHandler: Creating database path in external directory s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,434 INFO [Metastore-Handler-Pool: Thread-1338] HMSHandler: Created database path in external directory s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,438 INFO [Metastore-Handler-Pool: Thread-1334] HMSHandler: Created database path in external directory s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> h3. Sentinel and cleanup
> A watcher observed the database directory during the race window and uploaded a sentinel object below it:
> {code:java}
> WATCHER_START 2026-08-27T02:23:41Z
> DIRECTORY_VISIBLE 2026-08-27T02:23:48Z s3://warehouse/hms_direct_race_20260827T022340Z.db
> 46 bytes -> `minio/warehouse/hms_direct_race_20260827T022340Z.db/sentinel/metadata/race-sentinel.txt`
> SENTINEL_UPLOAD_DONE 2026-08-27T02:23:48Z
> WATCHER_END 2026-08-27T02:23:48Z
> {code}
> The unique-key conflict and subsequent cleanup appeared together in the Hive Metastore logs:
> {code:java}
> Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> at org.apache.hadoop.hive.metastore.HMSHandler.create_database_core(HMSHandler.java:1274)
> at org.apache.hadoop.hive.metastore.HMSHandler.create_database(HMSHandler.java:1343)
> 2026-08-27T02:23:58,482 WARN [Metastore-Handler-Pool: Thread-1342] FileUtils: No such file or directory: s3a://warehouse/hms_direct_race_20260827T022340Z.db; Force to delete it.
> 2026-08-27T02:23:58,487 ERROR [Metastore-Handler-Pool: Thread-1341] FileUtils: Failed to delete s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> After the test, no objects remained under the original path or the corresponding Trash path:
> {code:java}
> === OBJECTS UNDER DIRECT PATH ===
> === OBJECTS UNDER TRASH PATH ===
> === DIRECT PATH STAT ===
> mc: <ERROR> Unable to stat `minio/warehouse/hms_direct_race_20260827T022340Z.db/`. Object does not exist.
> {code}
> At the same time, the database row still existed in the MySQL {{DBS}} table:
> {code:java}
> DB_ID NAME CTLG_NAME DB_LOCATION_URI
> 104 hms_direct_race_20260827t022340z hive s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> Trash was disabled in this standalone reproduction, so cleanup directly deleted the directory instead of moving it to {{{}.Trash/Current{}}}.
> h2. Actual Result
> A failed {{create_database}} request may recursively clean up the entire final database directory rather than cleaning up only request-private resources.
> This can leave the system in the following state:
> * Database metadata still exists in Hive Metastore.
> * Table metadata may still exist in Hive Metastore.
> * Iceberg {{metadata_location}} values still reference the original object-storage paths.
> * The corresponding Iceberg metadata files have been moved to Trash or recursively deleted.
> * Subsequent Iceberg queries, startup recovery, or metadata commits receive 404/not-found errors.
> * Iceberg tables become unreadable or unwritable.
> h2. Expected Result
> When multiple requests concurrently create the same database:
> * One request should successfully create the database.
> * Other requests should receive {{AlreadyExistsException}} or an equivalent database-already-exists error.
> * Failed requests must not delete, move, or otherwise modify the shared final database directory.
> * Failed requests may only recursively clean up resources that are provably private to that request, such as a request-scoped staging path.
> h2. Root Cause
> {{create_database}} uses a non-atomic check-then-create sequence:
> {code:java}
> get_database_core(...);
> if (database does not exist) {
> create_database_core(...);
> }
> {code}
> Inside {{{}create_database_core{}}}, filesystem ownership is inferred from logic similar to:
> {code:java}
> if (!wh.isDir(dbPath)) {
> if (!wh.mkdirs(dbPath)) {
> throw new MetaException(...);
> }
> madeDir = true;
> }
> {code}
> A successful {{mkdirs}} call only shows that the directory became available through this request. It does not prove continuing exclusive ownership. Multiple requests may set {{madeDir}} to {{{}true{}}}, and even a directory created exclusively by one request may subsequently be adopted by a different request that wins the metadata transaction.
> When one request commits the Metastore metadata and other requests fail because of {{{}DBS.UNIQUE_DATABASE{}}}, the failed requests execute cleanup similar to:
> {code:java}
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(dbPath, true);
> }
> }
> {code}
> The corresponding external database path cleanup in Hive {{4.0.0-beta-1}} is equivalent to:
> {code:java}
> if (madeExternalDir && dbExtPath != null) {
> wh.deleteDir(dbExtPath, true, db);
> }
> {code}
> The cleanup scope is the entire final database directory, not an empty request-private directory or an object written only by the failed request.
> The unique-key conflict itself is an expected concurrency outcome. The data-loss risk comes from treating local directory-creation provenance as authorization to recursively delete a shared final location after metadata rollback.
> h2. Source References
> * Hive {{4.0.0-beta-1}} {{{}HMSHandler.create_database_core{}}}: [source|https://github.com/apache/hive/blob/rel/release-4.0.0-beta-1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L1138-L1341]
> * Hive {{3.1.2}} {{{}create_database_core{}}}: [source|https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1240-L1287]
> * Hive {{4.2.1}} {{{}HMSHandler.create_database_core{}}}: [source|https://github.com/apache/hive/blob/rel/release-4.2.1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L1163-L1347]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)